# gizml:llm language models, fast on any hardware. A model family is a recipe: its config.json read as an Arch, and the names of its tensors. Every family runs on the same operations, so a new family is one recipe. `convert` turns a Hugging Face folder into this module's format once: each linear is stored in a format (Formats), after a Hadamard rotation of each block of 128 inputs, in the order the GPU reads it. `load` opens a converted model on the best device it finds, and its Engine runs steps of many sequences at once. ```gizlang import * as Llm from "gizml:llm"; ``` ## Types ### `Message` ```gizlang type Message = { role: string; content: string; reasoning?: string; calls?: Vec; } ``` A message of a chat: its role ("system", "user", "assistant" or "tool") and text; an assistant's reasoning before its answer and the tools it called. - `role`: `string` - `content`: `string` - `reasoning`: `_gizlang_sdk_core.Option` - `calls`: `_gizlang_sdk_core.Option>` ### `Arch` ```gizlang type Arch = { family: string; vocab: i64; dim: i64; ffn: i64; layers: i64; kinds: Vec; eps: f64; heads: i64; kv: i64; headDim: i64; rotary: i64; rope: Vec; qkNorm: bool; gated: bool; bias: bool; kHeads: i64; vHeads: i64; kDim: i64; vDim: i64; conv: i64; tied: bool; mtp: i64; context: i64; prefix: string; sections: Vec; hc: i64; hcRank: i64; experts: i64; topK: i64; ple: i64; eos: i64; sigmoidGate: bool; } ``` What a model computes, as its family's recipe reads config.json. Attention: `heads` query heads of `headDim` over `kv` key-value heads, rotary positions on the first `rotary` dimensions of a head (`rope` their inverse frequencies), RMS norms of q and k (`qkNorm`), a sigmoid gate on the output (`gated`), biases on q, k and v. Recurrent layers: `kHeads` query-key heads of `kDim`, `vHeads` value heads of `vDim`, a causal convolution of `conv` taps. `mtp` layers predict a further token. - `family`: `string` - `vocab`: `i64` - `dim`: `i64` - `ffn`: `i64` - `layers`: `i64` - `kinds`: `Vec` - `eps`: `f64` - `heads`: `i64` - `kv`: `i64` - `headDim`: `i64` - `rotary`: `i64` - `rope`: `Vec` - `qkNorm`: `bool` - `gated`: `bool` - `bias`: `bool` - `kHeads`: `i64` - `vHeads`: `i64` - `kDim`: `i64` - `vDim`: `i64` - `conv`: `i64` - `tied`: `bool` - `mtp`: `i64` - `context`: `i64` - `prefix`: `string` - `sections`: `Vec` - `hc`: `i64` - `hcRank`: `i64` - `experts`: `i64` - `topK`: `i64` - `ple`: `i64` - `eos`: `i64` - `sigmoidGate`: `bool` ### `Bits` ```gizlang type Bits = { layers: i64; head: i64; } ``` The format of each kind of linear (Formats): `layers` for the layers, `head` for the output head. - `layers`: `i64` - `head`: `i64` ### `Model` ```gizlang type Model = { arch: Arch; folder: string; tokenizer: Tokenizer; stops: Vec; template: string; system: string; think: bool; calls: bool; efforts: Vec; sampling: Sampling; embed: Stored; layers: Vec; norm: i64; head: i64; mtp: Option; mixer: Vec; linears: Vec; small: Vec>; gpu: Option; device: Vec; deviceSmall: Vec; signs: Vec; files: FlashFiles; vision?: Vision; stages: Vec; npu?: NpuLlm.Model; } ``` A converted model, on the GPU that gizhw:gpu opens (GIZLANG_DEVICE) when there is one (`device`), else on the host (`linears`); the small tensors are on both. The embedding stays in the file: a step reads its tokens' rows. `template` is the chat layout ("chatml", "llama3" or ""), `system` the system turn it writes when a chat has none, `think` whether its assistant turn opens with a thinking block, `calls` whether it writes tool calls as ...; `efforts` the instructions its template gives a thinking turn at low, medium and high reasoning effort ("": none); `stops` are the tokens that end a generation and `sampling` the model's own settings. `mtp` is the MTP layer on a GPU when the model has one, `vision` its vision tower when its folder has one (convertVision), `stages` the models of its later layers on other GPUs (loadAcross). - `arch`: `M23.Arch` - `folder`: `string` - `tokenizer`: `M34.Tokenizer` - `stops`: `Vec` - `template`: `string` - `system`: `string` - `think`: `bool` - `calls`: `bool` - `efforts`: `Vec` - `sampling`: `M23.Sampling` - `embed`: `gizlang_safetensors.Stored` - `layers`: `Vec` - `norm`: `i64` - `head`: `i64` - `mtp`: `_gizlang_sdk_core.Option` - `mixer`: `Vec` - `linears`: `Vec` - `small`: `Vec>` - `gpu`: `_gizlang_sdk_core.Option` - `device`: `Vec` - `deviceSmall`: `Vec` - `signs`: `Vec` - `files`: `M23.FlashFiles` - `vision`: `_gizlang_sdk_core.Option` - `stages`: `Vec` - `npu`: `_gizlang_sdk_core.Option` ### `ChatTokens` ```gizlang type ChatTokens = { tokens: Vec; mark: i64; head: i64; } ``` A chat's prompt as tokens (Model.chatPrompt), the mark of Engine.add (where its messages end, before the assistant's turn starts: the chat's next request starts with these messages, so its slot reads only the rest) and its head (where the system turn with the tools ends: the chats with the same system turn share its keys and values). - `tokens`: `Vec` - `mark`: `i64` - `head`: `i64` ### `Engine` ```gizlang type Engine = { model: Model; slots: i64; context: i64; rows: i64; lengths: Vec; keys: Vec>; values: Vec>; convs: Vec>; states: Vec>; replay: Vec>; hidden: Vec; work: Vec; pages: Vec>; table: Vec; free: Vec; maxPages: i64; seen: Vec; shapes: Vec; recordings: Vec; depth: i64; pending: Vec; drafts: i64; kept: i64; draftBlocks: Vec; blockEntry: Vec; nextEntry: i64; paces: Vec; stepNs: f64; rowNs: f64; kv: KvFormat; waiting: Vec; turns: Vec; next: Vec; guesses: Vec>; held: Vec>; freed: Vec; saved: Vec; pieces: Vec; shared: Vec; borrowed: Vec; stored: Vec; ids: i64; steps: i64; reuse: bool; flash: Flash; stages: Vec; } ``` The engine of a model: `slots` sequences of at most `context` positions, steps of at most `rows` rows. - `model`: `M23.Model` - `slots`: `i64` - `context`: `i64` - `rows`: `i64` - `lengths`: `Vec` - `keys`: `Vec>` - `values`: `Vec>` - `convs`: `Vec>` - `states`: `Vec>` - `replay`: `Vec>` - `hidden`: `Vec` - `work`: `Vec` - `pages`: `Vec>` - `table`: `Vec` - `free`: `Vec` - `maxPages`: `i64` - `seen`: `Vec` - `shapes`: `Vec` - `recordings`: `Vec` - `depth`: `i64` - `pending`: `Vec` - `drafts`: `i64` - `kept`: `i64` - `draftBlocks`: `Vec` - `blockEntry`: `Vec` - `nextEntry`: `i64` - `paces`: `Vec` - `stepNs`: `f64` - `rowNs`: `f64` - `kv`: `M23.KvFormat` - `waiting`: `Vec` - `turns`: `Vec` - `next`: `Vec` - `guesses`: `Vec>` - `held`: `Vec>` - `freed`: `Vec` - `saved`: `Vec` - `pieces`: `Vec` — The bytes of each token (Tokenizer.tokenBytes), read at the first request with a grammar. - `shared`: `Vec` - `borrowed`: `Vec` - `stored`: `Vec` - `ids`: `i64` - `steps`: `i64` - `reuse`: `bool` - `flash`: `M23.Flash` - `stages`: `Vec` ### `Visual` ```gizlang type Visual = { rows: Vec; embeds: Vec; deep: Vec>; grid: Vec; } ``` Rows of a step that hold no token's embedding (Qwen3-VL's pictures): `rows` of the step, their embeddings (dim floats a row, in turn), `deep` the features added to them after each of the first layers (Qwen3-VL's deepstack, rows as embeds), and `grid` every row's three rotary positions (time, height, width; a text row's three are its position: see ropeTable). - `rows`: `Vec` - `embeds`: `Vec` - `deep`: `Vec>` - `grid`: `Vec` ### `KvFormat` ```gizlang type KvFormat = { keys: i64; values: i64; } ``` How an engine keeps keys and values: the bits of a key and of a value. 16: F16 values; 8, 4 or 2: codes of the head rotated by blocks of kvBlock, at 2^bits even steps (kvStore). - `keys`: `i64` - `values`: `i64` ### `ExpertCache` ```gizlang type ExpertCache = { slots: i64; hits: i64; misses: i64; readMs: f64; } ``` The expert cache of an engine: its slots, the experts found in it and read into it, and the ms the reads took. - `slots`: `i64` - `hits`: `i64` - `misses`: `i64` - `readMs`: `f64` ### `Guess` ```gizlang type Guess = { token: i64; q: Dist; } ``` A draft per sequence: its token and the distribution it was drawn from (all on the token when it is the most likely). - `token`: `i64` - `q`: `M23.Dist` ### `Vision` ```gizlang type Vision = { named: Map; gpu: Option; deep: Vec; depth: i64; width: i64; heads: i64; side: i64; } ``` A Qwen vision tower (loadVision): its weights by name where it runs (linears BF16, the rest f32; the patch embedding f32 [width, 768], the position table on the host), its blocks, width and heads, the side of its position table, and the blocks after which a deepstack merger reads the stream. - `named`: `_gizlang_sdk_collections.Map` - `gpu`: `_gizlang_sdk_core.Option` - `deep`: `Vec` - `depth`: `i64` - `width`: `i64` - `heads`: `i64` - `side`: `i64` ### `Stream` ```gizlang type Stream = { ids: Vec; tags: Vec; grid: Vec; visual: Visual; shift: i64; } ``` A prompt as a language model with a vision tower reads it: its token ids, each with three rotary positions (`grid`: its index plus `shift`, which a picture's rows lower, as their grid is smaller than their count) and a `tag` of the caller's, and the pictures' rows (`visual`: the tower's rows at their places). - `ids`: `Vec` - `tags`: `Vec` - `grid`: `Vec` - `visual`: `M23.Visual` - `shift`: `i64` ### `Shown` ```gizlang type Shown = { picture: Tensor; times: Vec; } ``` What a chat shows (Vision.chat): a picture ([3, h, w] f32 in [0, 1] on the host, h and w multiples of 32), or a video's frames ([n, 3, h, w]) with each frame's second in `times`. - `picture`: `M32.Tensor` - `times`: `Vec` ### `Seen` ```gizlang type Seen = { chat: ChatTokens; visual: Visual; } ``` A chat's prompt with what it shows (Vision.chat): its tokens, mark and head, and the Visual of the whole prompt (Request.visual). - `chat`: `M23.ChatTokens` - `visual`: `M23.Visual` ### `Sampling` ```gizlang type Sampling = { temperature: f64; topK: i64; topP: f64; seed: u64; presence?: f64; frequency?: f64; repetition?: f64; grammar?: Grammar; after?: i64; logprobs?: i64; } ``` How tokens are picked: temperature 0 the most likely, else one of the topK most likely (0: any) within topP of their probability, at temperature. Before that the logits are shaped, as OpenAI and vLLM shape them: less by `presence` for a token the answer has and by `frequency` for each time it has it; divided by `repetition` (above 0, times it below 0) for a token the prompt or the answer has. `grammar`: only the tokens it allows (gizml:grammar), from the token after `after` on (a reasoning model's end of thinking; none: from the first); a stop token only where the text is whole. `logprobs`: each token's log probability and the most likely ones (that many) are given, of the model's logits before shaping. A shaped request makes no drafts. - `temperature`: `f64` - `topK`: `i64` - `topP`: `f64` - `seed`: `u64` - `presence`: `_gizlang_sdk_core.Option` - `frequency`: `_gizlang_sdk_core.Option` - `repetition`: `_gizlang_sdk_core.Option` - `grammar`: `_gizlang_sdk_core.Option` - `after`: `_gizlang_sdk_core.Option` - `logprobs`: `_gizlang_sdk_core.Option` ### `Dist` ```gizlang type Dist = { ids: Vec; weights: Vec; } ``` What a sampled token is drawn from: tokens, most likely first, and their weights (a token's probability is its weight over their sum). - `ids`: `Vec` - `weights`: `Vec` ### `Request` ```gizlang type Request = { prompt: Vec; limit: i64; sampling: Sampling; visual?: Visual; } ``` A prompt to continue: at most `limit` new tokens. With `visual` some of its rows are pictures' (Vision.chat): their embeddings replace their tokens', and its rows take their three rotary positions from visual.grid (three for each of its first rows, the last a text row's), the rows after them their position plus that last row's shift. - `prompt`: `Vec` - `limit`: `i64` - `sampling`: `M23.Sampling` - `visual`: `_gizlang_sdk_core.Option` ### `Logprob` ```gizlang type Logprob = { token: i64; logprob: f64; top: Vec; tops: Vec; } ``` A token's log probability and the `top` most likely tokens with theirs, of the model's logits. - `token`: `i64` - `logprob`: `f64` - `top`: `Vec` - `tops`: `Vec` ### `Progress` ```gizlang type Progress = { id: i64; tokens: Vec; cached: i64; end: string; logprobs: Vec; } ``` What a step gave a request (Engine.advance): its new tokens (without a stop token), the tokens at the start of its prompt that its slot held already (not read again), and why it ended: "stop" (a stop token), "length" (its limit, the context's end, or a KV cache too small for it alone) or "" (it goes on). - `id`: `i64` - `tokens`: `Vec` - `cached`: `i64` - `end`: `string` - `logprobs`: `Vec` ### `Turn` ```gizlang type Turn = { id: i64; request: Request; random: Vec; made: i64; mark: i64; head: i64; cached: i64; reading?: Recognizer; } ``` A request in an engine's turn: its id, its random state, the tokens it gave, the prompt position after which its slot saves the recurrent states (Engine.add), the position after which its prompt's start is shared (Engine.add; 0: none), the prompt tokens its slot held already and its grammar's recognizer while the grammar holds (Sampling). - `id`: `i64` - `request`: `M23.Request` - `random`: `Vec` - `made`: `i64` - `mark`: `i64` - `head`: `i64` - `cached`: `i64` - `reading`: `_gizlang_sdk_core.Option` ### `Saved` ```gizlang type Saved = { at: i64; conv: Vec; state: Vec; } ``` A slot's recurrent states after its first `at` tokens (0: none): the convolutions' last inputs and the delta rules' states, on the host (from a GPU the states' BF16, two to a float). - `at`: `i64` - `conv`: `Vec` - `state`: `Vec` ### `Prefix` ```gizlang type Prefix = { tokens: Vec; saved: Saved; pages: Vec; keys: Vec>; values: Vec>; used: i64; stages: Vec; } ``` The start of a chat that left its slot for another (see admit): its tokens, the recurrent states after them, and their keys and values (on a GPU the bytes of their KV pages, on the host per pool layer); `used` is the step it was stored at. A later prompt that starts with the tokens takes them back into a slot and reads only the rest. On a model in stages, `stages` holds each later stage's pages and states (Engine.stages). - `tokens`: `Vec` - `saved`: `M23.Saved` - `pages`: `Vec` - `keys`: `Vec>` - `values`: `Vec>` - `used`: `i64` - `stages`: `Vec` ### `Shared` ```gizlang type Shared = { tokens: Vec; saved: Saved; pages: Vec; keys: Vec>; values: Vec>; used: i64; stages: Vec; } ``` A start that many prompts share (as the system turn and tools of an agent's chats, Model.chatTokens): its tokens (whole KV pages), the recurrent states after them, and their keys and values: on a GPU the KV pages of the pool that hold them, which the slots that start with them read in place (Engine.borrowed), on the host a copy per pool layer; `used` is the step a slot last took it. On a model in stages, `stages` holds each later stage's pages and states. - `tokens`: `Vec` - `saved`: `M23.Saved` - `pages`: `Vec` - `keys`: `Vec>` - `values`: `Vec>` - `used`: `i64` - `stages`: `Vec` ### `BertConfig` ```gizlang type BertConfig = { kind: string; layers: i64; heads: i64; dim: i64; eps: f64; act: string; positions: i64; offset: i64; } ``` A BERT-family encoder as a Hugging Face config.json gives it: BERT, RoBERTa and XLM-RoBERTa have one layout and differ in where positions start (`offset`: 0 for BERT, pad_token_id + 1 for RoBERTa). act is hidden_act: "gelu" (the erf form), "relu" or a tanh form ("gelu_new", "gelu_pytorch_tanh"). - `kind`: `string` - `layers`: `i64` - `heads`: `i64` - `dim`: `i64` - `eps`: `f64` - `act`: `string` - `positions`: `i64` - `offset`: `i64` ### `ModelFiles` ```gizlang type ModelFiles = (path: string) => Result, string> ``` Reads the file at a path of a model's repository: none when the model has no such file. ### `Encoder` ```gizlang type Encoder = { config: BertConfig; weights: Map; tokenizer: Tokenizer; pooling: string; normalize: bool; limit: i64; } ``` A sentence encoder as sentence-transformers 3 builds it from a model's files: an encoder, then the pooling of its token states ("mean", "cls", "max", "mean_sqrt_len" or "last"), then L2 normalization when the model has that module. - `config`: `M23.BertConfig` - `weights`: `_gizlang_sdk_collections.Map` - `tokenizer`: `M34.Tokenizer` - `pooling`: `string` - `normalize`: `bool` - `limit`: `i64` ### `CrossEncoder` ```gizlang type CrossEncoder = { config: BertConfig; weights: Map; tokenizer: Tokenizer; head: string; labels: i64; activation: string; limit: i64; } ``` A cross-encoder as sentence-transformers' CrossEncoder builds it: a sequence classification model that reads a pair of texts (a query and a document) and gives the scores of its labels. `head` is "bert" (BertForSequence- Classification: the pooler's tanh on the first token, then the classifier) or "roberta" (RobertaFor- and XLMRobertaForSequenceClassification: classifier.dense, tanh, classifier.out_proj). `activation` is the function on the scores: "sigmoid" (the default with one label), "identity" (the default with more) or "tanh". - `config`: `M23.BertConfig` - `weights`: `_gizlang_sdk_collections.Map` - `tokenizer`: `M34.Tokenizer` - `head`: `string` - `labels`: `i64` - `activation`: `string` - `limit`: `i64` ### `Ranked` ```gizlang type Ranked = { corpus: i64; score: f64; } ``` A document of a ranking: its index in the documents and its score. - `corpus`: `i64` - `score`: `f64` ## Functions ### `archOf` ```gizlang function archOf(config: JsonValue): Arch throws string ``` The Arch of a model by its config.json; an error names what this module does not run yet. Effects: fail. ### `convert` ```gizlang function convert(sys: System, from: string, to: string, bits: Bits, layers: i64 = 0, lora: string = ""): i64 throws string ``` Converts the Hugging Face model in folder `from` (config.json, safetensors, tokenizer), or the GGUF file `from` (gizlang:gguf: Llama, Qwen2 and Qwen3, its block types read as they are; the embedding keeps its block type), to this module's format in folder `to`: model.safetensors with every linear quantized (see Quantization), the norms and small tensors as f32 and the embedding as its source holds it, with the source's config, tokenizer and generation files. With `layers` less than the model's, only its first `layers` layers, without the final norm, the head and the MTP layer (a source cut after them, as MiniMax H3's text encoder, holds no more): load it with the same `layers`. A cut Qwen3.8 Flash keeps its mixer and head (a test of its first layers). A vision tower (model.visual.*) goes to visual.safetensors (convertVision). Returns the bytes written. With `lora`, the PEFT LoRA adapter in that folder (adapter_config.json, adapter_model.safetensors) is merged into the linears it adapts before they are quantized: W + alpha / r B A (alpha / sqrt(r) with rsLoRA). Effects: fail, read, write, console, io. ### `load` ```gizlang function load(sys: System, folder: string, onGpu: bool = true, layers: i64 = 0): Model throws string ``` The converted model in `folder`, on the GPU when `onGpu` and there is one, and across the driver's other GPUs too when its weights do not leave SPLIT_ROOM free on that one (loadAcross). With `layers` less than the model's, only its first `layers` layers, without the final norm, the head and the MTP layer: an engine of it gives hidden states (Engine.hidden), not logits. A gizml:npullm folder (model.q4nx) runs on this host's NPU (Model.npu). Effects: fail, read, write, console, io. ### `loadAcross` ```gizlang function loadAcross(sys: System, folder: string, gpus: Vec, counts: Vec = []): Model throws string ``` The converted model in `folder` across `gpus` (Gpu.others), in stages of consecutive layers: the first GPU holds the first stage (and the vision tower), the last the last stage with the final norm and the head (Model.stages). A stage's layer count is counts[i], by default in proportion to its GPU's free memory. A step runs the stages in turn, each on its rows' hidden states after the one before (Engine). Without the MTP layer, and not for Qwen3.8 Flash yet. The first GPU is current at the end. Effects: fail, read, write, console, io. ### `loadOn` ```gizlang function loadOn(sys: System, folder: string, gpu: Option, layers: i64 = 0, from: i64 = 0, streamed: bool = false): Result ``` `load` on the GPU `gpu` (none: the host). With `from`, the `layers` layers from layer `from` on (a stage of loadAcross): the norm, the head and the MTP layer only where they are the model's last. `streamed`: the vision tower's linears stay in host memory (loadVision). Effects: fail, read, write, console, io. ### `converted` ```gizlang function converted(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string throws string ``` The folder of the model in folder `folder` ("" for its top) of Hugging Face repository `repo` ("Qwen/Qwen3.5-4B") converted with `bits` (and `layers`, as `convert` cuts), for `load` with the same `layers`: the folder is downloaded through `hub` once (Hub.snapshot) and converted once into `.gizlang-llm/REPO[/FOLDER]/BITS` in the hub's folder. A conversion is written to a partial folder and renamed when it is complete, so a stopped one is done again, never loaded. Effects: fail, read, write, console, io. ### `convertedFolder` ```gizlang function convertedFolder(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string ``` Where `converted` puts a conversion; the folder holds model.safetensors once the conversion is complete. Effects: fail. ### `toolCalls` ```gizlang function toolCalls(text: string, tools: Vec, first: i64 = 0): Vec ``` The tool calls in the text a model wrote after its answer (Model.calls: VALUE...), each value as JSON by the type its tool gives the parameter: a string stays text, another type is read as JSON (text when it is not). Their ids are call_FIRST_K. Effects: fail. ### `kvFormat` ```gizlang function kvFormat(name: string): KvFormat throws string ``` The format a name gives: BITS for keys and values, or KEYS/VALUES ("4/2"); "" is the engine's choice (keys 0). Effects: fail. ### `kvName` ```gizlang function kvName(f: KvFormat): string ``` The name of a format, as kvFormat reads it. Effects: fail. ### `convertVision` ```gizlang function convertVision(sys: System, stored: Vec, config: JsonValue, to: string): i64 throws string ``` Writes the vision tower of the Hugging Face folder `from` (model.visual.*, as stored) to `to`/visual.safetensors, with its heads and deepstack blocks (config.json's vision_config) in the header; returns its bytes. Effects: fail, io. ### `hasVision` ```gizlang function hasVision(sys: System, folder: string): bool ``` Whether `folder` holds a vision tower (convertVision's). Effects: fail, io. ### `loadVision` ```gizlang function loadVision(sys: System, folder: string, gpu: Option, streamed: bool = false): Vision throws string ``` The vision tower of `folder`/visual.safetensors (convertVision's) on `gpu` when there is one. A file of format 1 (before 10/22: Qwen3-VL-32B's for H3, Qwen3.5-9B's for FLUX.3's upsampler) has 16 heads, and its deepstack mergers read blocks 8, 16 and 24. With `streamed`, its linears' weights stay in host memory and a pass uploads each as it reads it (Vision.w): the card keeps their bytes free between pictures, and a pass moves them over the bus. Effects: fail, read, write, console, io. ### `stream` ```gizlang function stream(deep: i64): Stream ``` An empty stream whose pictures keep `deep` deepstack features each (Vision.seen). Effects: fail. ### `visionSize` ```gizlang function visionSize(width: i64, height: i64, least: i64, most: i64): Vec ``` The size [w, h] at which Qwen's processors show a picture of `width` x `height` (smart_resize): each side a multiple of 32 (2 x 2 patches of 16), the area from `least` to `most` pixels, the aspect kept as near as they allow. Effects: fail. ### `greedy` ```gizlang function greedy(): Sampling ``` The most likely token at each step: the same tokens on each run. ### `pick` ```gizlang function pick(logits: Vec, at: i64, vocab: i64, s: Sampling, state: mut Vec): i64 ``` A token from logits[at, at + vocab); `state` holds the random state. Effects: fail. ### `bertWeights` ```gizlang function bertWeights(tensors: Vec): Map ``` The tensors of a BertModel or RobertaModel checkpoint by their names, without the "bert." or "roberta." prefix of a task model, and with LayerNorm.gamma and beta (older checkpoints) as weight and bias. Effects: fail. ### `bert` ```gizlang function bert(c: BertConfig, w: Map, ids: Vec, types: Vec, sequences: i64): Tensor throws string ``` The last hidden states (sequences tokens, dim) of the encoder for `sequences` sequences of one length: their token ids and token types one sequence after another, every token attended (no padding). The weights may be on a GPU (Encoder.on); the states are where the weights are. Effects: fail, read, write, console, io. ### `encoderOf` ```gizlang function encoderOf(files: ModelFiles): Encoder throws string ``` The encoder of a model by the files it reads from its repository: config.json, tokenizer.json, model.safetensors and, for a sentence-transformers model, modules.json, sentence_bert_config.json and the pooling module's config.json. Without modules.json the tokens are mean-pooled and not normalized, as sentence-transformers does. Effects: fail, read, write, console, io. ### `crossEncoderOf` ```gizlang function crossEncoderOf(files: ModelFiles): CrossEncoder throws string ``` The cross-encoder of a model by its repository's files: config.json (its sentence_transformers.activation_fn, when present, names the activation), tokenizer.json and model.safetensors. Effects: fail, read, write, console, io. ## Methods ### `Arch.qRows` ```gizlang function Arch.qRows(self: Arch): i64 ``` Values of a query (and of its output gate when the model has one) per token. Effects: fail. ### `Arch.qkvRows` ```gizlang function Arch.qkvRows(self: Arch): i64 ``` Rows of the fused q, k, v projection. Effects: fail. ### `Arch.convDim` ```gizlang function Arch.convDim(self: Arch): i64 ``` Channels of a recurrent layer's convolution: q, k and v. Effects: fail. ### `Arch.inRows` ```gizlang function Arch.inRows(self: Arch): i64 ``` Rows of a recurrent layer's fused input projection: q, k, v, the gate z, then b and a (one a value head). Effects: fail. ### `Model.chatPrompt` ```gizlang function Model.chatPrompt(self: Model, messages: Vec, thinking: bool, tools: Vec = [], effort: string = "high"): string ``` The prompt of a chat as the model's template writes it, ending with the start of the assistant's turn (with an empty thinking block unless `thinking`), a thinking turn with the instruction of its reasoning `effort` (low, medium or high: Model.efforts), and the tools it may call (OpenAI's tool objects, for a model whose template writes tool calls: Model.calls). Effects: fail. ### `Model.chatTokens` ```gizlang function Model.chatTokens(self: Model, messages: Vec, thinking: bool, tools: Vec = [], effort: string = "high"): ChatTokens ``` The tokens of `messages` in the model's chat template, up to where the assistant's turn starts (ChatTokens). Effects: fail. ### `Model.engine` ```gizlang function Model.engine(self: Model, slots: i64, context: i64, reserve: i64 = 134_217_728, drafts: i64 = 3, rows: i64 = 0, kv: string = ""): Result ``` An engine for `slots` sequences of up to `context` positions, reading up to `rows` rows a step (0: 256; at least 1 + drafts rows a slot, 2 without drafts). On a GPU the KV pages take the device memory left after the work buffers and recurrent states, at most what the slots can hold, keeping `reserve` bytes free. Keys and values take the format `kv` names (kvFormat; F16 where the heads take no codes, see kvWidth), by default the first of kvFormats in which the slots' whole contexts fit above 4 bits, or one whole context at most 4 bits, else the last (on the host 8 bits). An engine reads up to `drafts` drafts a sequence and step (0: none; 3 is DRAFTS), and with the model's MTP layer Engine.advance drafts them where that gives tokens faster (EXPLORE), on a GPU or the host (where 1 draft is the fastest: its 4-bit layers take two rows in the time of one, and each further draft costs a step of the MTP layer). Effects: fail, read, write, console, io. ### `Engine.reset` ```gizlang function Engine.reset(self: mut Engine, slot: i64): void ``` Empties a slot for a new sequence. Effects: fail, read, write, console, io. ### `Engine.length` ```gizlang function Engine.length(self: Engine, slot: i64): i64 ``` Positions held by a slot. Effects: fail. ### `Engine.step` ```gizlang function Engine.step(self: mut Engine, sys: System, tokens: Vec, slots: Vec, wanted: Vec, drafts: Vec = [], visual: Option = undefined): Vec throws string ``` Reads `tokens` into their slots (the rows of a slot together and in order, after what the slot holds) and returns the logits of the rows `wanted` (vocab floats each, in that order). On an engine that drafts (Engine.depth), a slot's last drafts[slot] rows (none when drafts is empty) are drafts of the tokens after its row before them: they wait for Engine.accept, which the slot's next step needs first. With `visual`, some rows are a picture's (Visual, as Engine.hidden), and every row takes its three rotary positions from it. Effects: fail, read, write, console, io. ### `Engine.accept` ```gizlang function Engine.accept(self: mut Engine, slot: i64, kept: i64): bool throws string ``` Keeps the first `kept` of a slot's drafts of its last step and drops the others. Effects: fail, read, write, console, io. ### `Engine.hidden` ```gizlang function Engine.hidden(self: mut Engine, sys: System, tokens: Vec, slots: Vec, layers: Vec = [], visual: Option = undefined): Vec throws string ``` Reads `tokens` into their slots as Engine.step does and returns the residual stream of each row after each of `layers` (counted from 1; none: the model's last), dim floats a row and layer: Hugging Face's hidden_states[l] for each l, before the final norm (Hugging Face norms the whole model's last hidden state). With `visual`, some rows are a picture's (Visual), not their tokens' embeddings. Effects: fail, read, write, console, io. ### `Engine.padding` ```gizlang function Engine.padding(self: mut Engine, sys: System, slot: i64, token: i64, position: i64, layers: Vec = []): Vec throws string ``` The hidden states (as Engine.hidden) of one row of `token` at rotary position `position` that sees the keys of the slot's tokens but not its own, and leaves the slot as it was: a padding row of a Hugging Face batch, whose mask hides the padding keys from every row. Every padding row of a sequence is this one row (FLUX.3 reads Qwen3-VL's, at position 1). Models of attention layers only. Effects: fail, read, write, console, io. ### `Engine.expertCache` ```gizlang function Engine.expertCache(self: Engine): ExpertCache ``` The expert cache counts of the engine. ### `Vision.seen` ```gizlang function Vision.seen(self: Vision, picture: Tensor, later: Vec = []): Vec ``` The vision tower's rows of a picture ([3, h, w] f32 in [0, 1] on the host, h and w multiples of 32): its merged rows ([h w / 1024, the language model's width]) and the deepstack features after each of its `deep` blocks (rows as the merged), f32 on the host. A video's pair of frames is `picture` and `later` (one frame). Effects: fail, read, write, console, io. ### `Stream.words` ```gizlang function Stream.words(self: mut Stream, tokens: Vec, tag: i64): void ``` The tokens `tokens` of `tag`. Effects: fail. ### `Stream.picture` ```gizlang function Stream.picture(self: mut Stream, vision: Vision, picture: Tensor, start: i64, pad: i64, end: i64, later: Vec = []): void ``` Picture ([3, h, w] f32 in [0, 1] on the host, h and w multiples of 32) as `vision` reads it: the token `start`, a `pad` token for each row of the tower's merged rows (on the rotary positions of its grid), then `end`. A video's pair of frames is `picture` and `later` (one frame; the tower's temporal patch of two). Effects: fail, read, write, console, io. ### `Stream.video` ```gizlang function Stream.video(self: mut Stream, vision: Vision, tok: Tokenizer, clip: Tensor, times: Vec, start: i64, pad: i64, end: i64, tag: i64): bool throws string ``` A video ([n, 3, h, w] f32 in [0, 1] on the host, h and w multiples of 32; `times`: each frame's second) as Qwen3-VL reads it: its frames in pairs (the last repeated when n is odd), each pair the text "" (`tok`'s tokens, of `tag`; t the second of its middle) and then the pair as Stream.picture shows it. Effects: fail, read, write, console, io. ### `Stream.part` ```gizlang function Stream.part(self: Stream, at: i64, end: i64, dim: i64): Option ``` The pictures' rows that fall in tokens `at` to `end`, as the step over them takes them (rows `dim` wide), or none when there are no pictures. Effects: fail. ### `Vision.chat` ```gizlang function Vision.chat(self: Vision, tok: Tokenizer, chat: ChatTokens, shown: Vec): Seen throws string ``` The prompt `chat` (Model.chatTokens) with what it shows: in its text each picture is Qwen's "<|vision_start|><|image_pad|><|vision_end|>" and each video "<|vision_start|><|video_pad|><|vision_end|>", in the order of `shown`; each becomes the tower's rows (Stream.picture, Stream.video), and the mark and head move past the rows before them. Effects: fail, read, write, console, io. ### `Engine.generate` ```gizlang function Engine.generate(self: mut Engine, sys: System, requests: Vec): Vec> throws string ``` Generates for every request and returns each one's new tokens (without the stop token): Engine.add of each, then Engine.advance until none is in turn. Effects: fail, read, write, console, io. ### `Engine.warm` ```gizlang function Engine.warm(self: mut Engine, sys: System): bool throws string ``` Runs once each kind of step a server meets first: a long prompt read in parts, then 1 to `slots` sequences that read a prompt and write tokens. The first run of a step shape compiles, tunes and records its kernels (seconds for a large model), so a server warms before it takes requests. Effects: fail, read, write, console, io. ### `Engine.add` ```gizlang function Engine.add(self: mut Engine, request0: Request, mark: i64 = 0, head: i64 = 0): i64 throws string ``` Puts a request in turn and returns its id, which Engine.advance's results name. A free slot that holds the start of its prompt (an earlier request's prompt and answer) takes it and reads only the rest; so does a slot that takes back such a start from the host, where a slot keeps what it held when another request takes it (Prefix). After `mark` tokens of the prompt (0: none) its slot also keeps the recurrent states, so a later prompt that starts with those tokens but not with all the slot read reads only the rest too; for a chat that is where its messages end (Model.chatTokens). The prompt's first `head` tokens (0: none; for a chat its system turn with the tools), cut to whole KV pages and when at least SHARED_MIN, become a start that later prompts share (Shared): a slot whose prompt starts with it reads only the rest, and on a GPU all such slots read one copy of its keys and values. Effects: fail. ### `Engine.cancel` ```gizlang function Engine.cancel(self: mut Engine, id: i64): bool ``` Takes a request out of turn; false when it is not in turn (it ended). Its slot keeps what it read. Effects: fail. ### `Engine.busy` ```gizlang function Engine.busy(self: Engine): bool ``` Whether requests are in turn. Effects: fail. ### `Engine.advance` ```gizlang function Engine.advance(self: mut Engine, sys: System): Vec throws string ``` One step of the requests in turn, and what each gained. First free slots take waiting requests (Engine.add). The step reads the next token of every generating sequence and, in the rows left, the next piece of the prompts not read yet. When the KV pool has no room for the step, a free slot gives its pages, else the request that came last waits again from where it is (its prompt and answer so far). A drafting engine (Engine.depth) also reads each generating sequence's drafts of the tokens after that: each row judges the draft after it (kept when it is the token picked; when sampling, kept with its probability and otherwise a token other than it is picked, which gives the same distribution as without drafts) up to the first rejected, and when all are kept the last row gives one more token, so a step reading weights once can give 1 + depth tokens. The step reads them as drafts (Engine.step) and keeps those judged right (Engine.accept). Then the MTP layer reads every row kept, with the token after it, and where drafts gave tokens faster for as many generating sequences (EXPLORE), drafts after each sequence's last; it reads each draft again, with its own hidden state for it, for the next. Effects: fail, read, write, console, io. ### `Engine.complete` ```gizlang function Engine.complete(self: mut Engine, sys: System, prompt: string, limit: i64, s: Sampling): string throws string ``` The text the model writes after `prompt` (special tokens in it read as such), at most `limit` tokens. Effects: fail, read, write, console, io. ### `Encoder.on` ```gizlang function Encoder.on(self: Encoder, gpu: Gpu): Encoder ``` The encoder with its weights on `gpu`: it encodes there and gives the same embeddings to f32 precision. Effects: fail, read, write, console, io. ### `Encoder.embed` ```gizlang function Encoder.embed(self: Encoder, text: string, normalize: bool): Vec throws string ``` The embedding of a text (Encoder.embedBatch of one). Effects: fail, read, write, console, io. ### `Encoder.embedBatch` ```gizlang function Encoder.embedBatch(self: Encoder, texts: Vec, normalize: bool, batch: i64): Vec> throws string ``` The embeddings of texts: their token states pooled and, when the model says so or `normalize`, of length 1. The texts of one token count go through the encoder together, at most `batch` of them at a time, so no token is padding. Effects: fail, read, write, console, io. ### `CrossEncoder.on` ```gizlang function CrossEncoder.on(self: CrossEncoder, gpu: Gpu): CrossEncoder ``` The cross-encoder with its weights on `gpu`. Effects: fail, read, write, console, io. ### `CrossEncoder.scores` ```gizlang function CrossEncoder.scores(self: CrossEncoder, query: string, documents: Vec, batch: i64): Vec> throws string ``` The scores of the labels of each pair (query, documents[i]), after the activation: CrossEncoder.predict. The pairs of one token count go through the encoder together, at most `batch` at a time. Effects: fail, read, write, console, io. ### `CrossEncoder.rank` ```gizlang function CrossEncoder.rank(self: CrossEncoder, query: string, documents: Vec, top: i64, batch: i64): Vec throws string ``` The documents by the score of (query, document), the highest first, at most `top` of them (0: all): CrossEncoder. rank. Equal scores keep the documents' order. The model must have one label. Effects: fail, read, write, console, io. ## Constants ### `ATTENTION` ```gizlang const ATTENTION = 0 ``` A layer of softmax attention over a KV cache. ### `RECURRENT` ```gizlang const RECURRENT = 1 ``` A layer of gated delta-rule linear attention over a recurrent state. ### `CALL` ```gizlang const CALL = "" ``` Where a model that writes tool calls (Model.calls) starts them. ### `DRAFTS` ```gizlang const DRAFTS = 3 ``` The tokens an engine drafts per sequence and step by default: the n-th draft is kept less often than the one before, and each costs a step of the MTP layer and the head.