GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/llm/state

the data of a model and of its engine: their records, the constants of their shapes, the pure reads over them and the reset of a slot. The modules that run an engine (kv, gemm, delta, gpu, host, flash, generate, engine) and the module that loads a model (model) import it. It imports none of them.

gizlang
import * as State from "gizai/llm/state";

Types

Engine

gizlang
type Engine = { model: Model; slots: i64; context: i64; rows: i64; lengths: Vec<i64>; long: Vec<bool>; kv: KvPlan; host: HostState; device: DeviceState; draft: Drafting; queue: Queue; prefix: PrefixCache; vocabulary: Vocabulary; flash: Flash; stages: Vec<Engine>; exact: bool; fuse: Vec<string>; lora: EngineLora; block?: BlockState; }

The engine of a model: slots sequences of at most context positions, steps of at most rows rows.

  • model: M70.Model
  • slots: i64
  • context: i64
  • rows: i64
  • lengths: Vec<i64>
  • long: Vec<bool>
  • kv: M65.KvPlan
  • host: M70.HostState
  • device: M70.DeviceState
  • draft: M70.Drafting
  • queue: M70.Queue
  • prefix: M70.PrefixCache
  • vocabulary: _gizlang_lib_ai__llm__grammar.Vocabulary — The tokens' bytes for grammars (Tokenizer.tokenBytes, sorted), read at the first request with a grammar.
  • flash: M70.Flash
  • stages: Vec<M70.Engine>
  • exact: bool — Linears give x to the GEMMs as the F32 the model computes (rounded to F16 there): no gemm16Kernel, whose 16-bit weights take x in their own type (BF16 for BF16 weights, as other servers' activations), and NV4 and F8 weights run as GEMVs, not GEMMs of x in NVFP4 or FP8. Off: the faster.
  • fuse: Vec<string> — The fusions the engine takes (EngineOptions.fuse, FUSIONS): conv, the delta rule's convolution within its chunks' terms (deltaPrepKernel), and nv, gate_up's activations as the down GEMM's NVFP4 input (gemmNvKernel's nv). Neither was faster on the RTX 5090 at 2048 rows, so both are off by default.
  • lora: M70.EngineLora — The LoRAs it serves (Engine.adapt): a request names one (Request.adapter).
  • block: _gizlang_sdk_core.Option<M70.BlockState>

BlockState

gizlang
type BlockState = { ring: Buffer; rope: Buffer; meta: Buffer; parts: Buffer; mixed: Buffer; from: Vec<i64>; high: Vec<i64>; }

A block drafter's state (Drafter): per slot its context keys and values of each drafter layer at the last positions (ring, F16: slot, layer, position modulo ringPositions, then the kv heads' keys and their values), the rotary table of the rows of a step (rope), a draft pass's tokens, sequences and rows (meta, blockMeta), its attention's partial results (parts) and the inputs of its linears (mixed). Per slot the first position its ring holds for the sequence there (from) and the end of the positions written to it (high): a draft reads positions from both on.

  • ring: M7.Buffer
  • rope: M7.Buffer
  • meta: M7.Buffer
  • parts: M7.Buffer
  • mixed: M7.Buffer
  • from: Vec<i64>
  • high: Vec<i64>

HostState

gizlang
type HostState = { keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; convs: Vec<Vec<f32>>; states: Vec<Vec<f32>>; replay: Vec<Vec<f32>>; hidden: Vec<f32>; }

An engine's state on the host: per slot and pool layer the keys and values (position, kv head, dimension; as kvStore keeps them, see kvRounded), per slot and recurrent layer the convolution's last inputs and the delta rule's state, and its last step's drafts' convolution inputs and delta rule terms (cpuReplay); the last hidden states that the MTP layer reads.

  • keys: Vec<Vec<f32>>
  • values: Vec<Vec<f32>>
  • convs: Vec<Vec<f32>>
  • states: Vec<Vec<f32>>
  • replay: Vec<Vec<f32>>
  • hidden: Vec<f32>

DeviceState

gizlang
type DeviceState = { work: Vec<Buffer>; pages: Vec<Vec<i64>>; table: Vec<f32>; free: Vec<i64>; maxPages: i64; seen: Vec<string>; shapes: Vec<string>; recordings: Vec<G.Recording>; seeing: i64; slack: i64; placed: i64; held: Ref<Vec<Buffer>>; taps: Ref<Vec<Buffer>>; deep: Ref<Vec<Buffer>>; }

An engine's state on a GPU: the work buffers (W_*), each slot's pages, the page table and the free pages; the step shapes launched once, the recordings of those launched again, the bytes a picture's pass takes (Vision.peak), those the driver may keep after passes (slack) and the buffer of both, held between passes (pictureRoom; a cell: the copies of a served engine share it, and a copy that kept its own Vec kept the buffer, 11/6).

  • work: Vec<M7.Buffer>
  • pages: Vec<Vec<i64>>
  • table: Vec<f32>
  • free: Vec<i64>
  • maxPages: i64
  • seen: Vec<string>
  • shapes: Vec<string>
  • recordings: Vec<M7.Recording>
  • seeing: i64
  • slack: i64
  • placed: i64
  • held: Ref<Vec<M7.Buffer>>
  • taps: Ref<Vec<M7.Buffer>> — The buffer of the hidden states a step keeps (Engine.hidden), of the engine's largest step: made at the first such step and held while the engine lives, so an encoder's prompts ask the card for no other.
  • deep: Ref<Vec<M7.Buffer>> — The buffers of a picture's rows for the deeper layers in such a step (Visual.deep), each of the largest step: held as taps.

Drafting

gizlang
type Drafting = { depth: i64; pending: Vec<i64>; bases: Vec<i64>; drafts: i64; kept: i64; draftBlocks: Vec<i64>; blockEntry: Vec<i64>; nextEntry: i64; steps: Vec<f64>; reach: Vec<f64>; ahead: Vec<Vec<Ahead>>; level: i64; forced: i64; lookup: i64; recall: i64; }

Drafts and pacing: the most drafts a sequence's step reads (0: none; see Engine.step), per slot its drafts of the last step not accepted yet and their first draft row (draftBase), and the drafts generate read and kept. The head blocks a draft reads (DRAFT_SHARE; empty: all; on a GPU also in W_DRAFT_BLOCKS), per head block its entry there or -1, and the entry of the second half the next new block takes. Per context class c (paceClass), count n of generating sequences and count i of draftLevels(depth), the ns a step took (a moving average; 0: not measured yet) at steps[(c * (slots + 1) + n) * levels + i]. Per depth i, the share of drafts right through draft i (reach[i], a moving average; -1: none judged yet), and per slot its drafts not judged yet (Ahead). The drafts of the last step and the count warming forces (-1: none; draftLevel). The tokens a lookup and a recalled draft match before them (EngineOptions.lookup, EngineOptions.recall).

  • depth: i64
  • pending: Vec<i64>
  • bases: Vec<i64>
  • drafts: i64
  • kept: i64
  • draftBlocks: Vec<i64>
  • blockEntry: Vec<i64>
  • nextEntry: i64
  • steps: Vec<f64>
  • reach: Vec<f64>
  • ahead: Vec<Vec<M70.Ahead>>
  • level: i64
  • forced: i64
  • lookup: i64
  • recall: i64

Ahead

gizlang
type Ahead = { at: i64; tokens: Vec<i64>; }

Drafts a source made after a sequence's next token, the first at index at of its held tokens: all it made, also those the step did not read, judged against the tokens the sequence holds later (LlmGenerate.judged).

  • at: i64
  • tokens: Vec<i64>

Queue

gizlang
type Queue = { waiting: Vec<Turn>; turns: Vec<Turn>; next: Vec<i64>; guesses: Vec<Vec<Guess>>; held: Vec<Vec<i64>>; freed: Vec<i64>; saved: Vec<Saved>; grams: Vec<Map<i64, i64>>; chains: Vec<Vec<i64>>; ids: i64; steps: i64; made: i64; read: i64; }

The requests in turn (Engine.add): those waiting for a slot, and per slot the one it serves (id -1: none), the token it reads next (-1: none yet, it reads its prompt) and its drafts of the tokens after that. Per slot the tokens its keys, values and states hold (a free slot keeps them for a prompt that starts with them), the step it was last freed at, the recurrent states it saved at a prompt's mark, and the index of its tokens' pairs (Queue.grams: the place after the latest of each pair; chains[i]: the place after the pair before place i, -1: none; recalled). The next request id and step.

  • waiting: Vec<M70.Turn>
  • turns: Vec<M70.Turn>
  • next: Vec<i64>
  • guesses: Vec<Vec<M70.Guess>>
  • held: Vec<Vec<i64>>
  • freed: Vec<i64>
  • saved: Vec<M70.Saved>
  • grams: Vec<_gizlang_sdk_collections.Map<i64, i64>>
  • chains: Vec<Vec<i64>>
  • ids: i64
  • steps: i64
  • made: i64
  • read: i64

PrefixCache

gizlang
type PrefixCache = { shared: Vec<Shared>; borrowed: Vec<i64>; stored: Vec<Prefix>; reuse: bool; folder: string; }

The prefix cache: the starts many prompts share (Shared), per slot how many of its first KV pages are a shared start's, and the starts of chats that left their slots, on the host or, past the host memory free, in files of folder (see Prefix; "?" until first used, "" for none). With reuse off each prompt reads all of its tokens.

  • shared: Vec<M70.Shared>
  • borrowed: Vec<i64>
  • stored: Vec<M70.Prefix>
  • reuse: bool
  • folder: string

Rows

gizlang
type Rows = { tokens: Vec<i64>; slots: Vec<i64>; positions: Vec<i64>; ropes: Vec<i64>; grid: Vec<i64>; own: i64; wanted: Vec<i64>; segments: Vec<i64>; drafts: Vec<i64>; layerInputs?: Vec<f32>; }

A step: per row its token, slot, position and rotary position (or three: grid, Visual.grid), the rows whose logits are wanted, per sequence its slot, first row and row count, and per sequence its drafts (its last rows, see Engine.generate). A row sees the keys before its position, and its own when own is 1 (0: a padding row, see Engine.padding).

  • tokens: Vec<i64>
  • slots: Vec<i64>
  • positions: Vec<i64>
  • ropes: Vec<i64>
  • grid: Vec<i64>
  • own: i64
  • wanted: Vec<i64>
  • segments: Vec<i64>
  • drafts: Vec<i64>
  • layerInputs: _gizlang_sdk_core.Option<Vec<f32>>

ExpertCache

gizlang
type ExpertCache = { slots: i64; hits: i64; misses: i64; readMs: f64; }

The expert cache of an engine: its slots, the experts found in it and read into it, and the ms the reads took.

  • slots: i64
  • hits: i64
  • misses: i64
  • readMs: f64

Guess

gizlang
type Guess = { token: i64; q: Dist; }

A draft per sequence: its token and the distribution it was drawn from (all on the token when it is the most likely).

  • token: i64
  • q: M72.Dist

Progress

gizlang
type Progress = { id: i64; tokens: Vec<i64>; cached: i64; end: string; logprobs: Vec<Logprob>; }

What a step gave a request (Engine.advance): its new tokens (without a stop token), the tokens at the start of its prompt that its slot held already (not read again), and why it ended: "stop" (a stop token), "length" (its limit, the context's end, or a KV cache too small for it alone) or "" (it goes on).

  • id: i64
  • tokens: Vec<i64>
  • cached: i64
  • end: string
  • logprobs: Vec<M72.Logprob>

Turn

gizlang
type Turn = { id: i64; request: Request; random: Vec<u64>; made: i64; mark: i64; head: i64; cached: i64; long: bool; since: i64; reading?: Recognizer; }

A request in an engine's turn: its id, its random state, the tokens it gave, the prompt position after which its slot saves the recurrent states (Engine.add), the position after which its prompt's start is shared (Engine.add; 0: none), the prompt tokens its slot held already, its grammar's recognizer while the grammar holds (Sampling), whether its sequence is long (stretches), and the ns (Task.nanos) it began to wait, or to run once it took a slot (admit).

  • id: i64
  • request: M72.Request
  • random: Vec<u64>
  • made: i64
  • mark: i64
  • head: i64
  • cached: i64
  • long: bool
  • since: i64
  • reading: _gizlang_sdk_core.Option<_gizlang_lib_ai__llm__grammar.Recognizer>

Saved

gizlang
type Saved = { at: i64; conv: Vec<f32>; state: Vec<f32>; pages: Vec<i64>; }

A slot's recurrent states after its first at tokens (0: none): the convolutions' last inputs and the delta rules' states. On a GPU they are in KV pages of the pool (pages, savedPages: a copy on the device, not over the bus), or on the host where the pool had no room or the start went to the host (Prefix); on the host conv and state (from a GPU the states' BF16, two to a float). One holder owns the pages: a copy moves them or copies them on the device.

  • at: i64
  • conv: Vec<f32>
  • state: Vec<f32>
  • pages: Vec<i64>

Prefix

gizlang
type Prefix = { tokens: Vec<i64>; long: bool; saved: Saved; pages: Vec<bytes>; keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; used: i64; stages: Vec<Prefix>; file: string; ple: Vec<f32>; }

The start of a chat that left its slot for another (see admit): its tokens, the recurrent states after them, and their keys and values (on a GPU the bytes of their KV pages, on the host per pool layer); used is the step it was stored at. A later prompt that starts with the tokens takes them back into a slot and reads only the rest. On a model in stages, stages holds each later stage's pages and states (Engine.stages). A long sequence's start (Turn.long) is taken back by a long one only: their keys turn by another table.

  • tokens: Vec<i64>
  • long: bool
  • saved: M70.Saved
  • pages: Vec<bytes>
  • keys: Vec<Vec<f32>>
  • values: Vec<Vec<f32>>
  • used: i64
  • stages: Vec<M70.Prefix>
  • file: string
  • ple: Vec<f32>

Shared

gizlang
type Shared = { tokens: Vec<i64>; long: bool; saved: Saved; pages: Vec<i64>; keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; used: i64; stages: Vec<Shared>; }

A start that many prompts share (as the system turn and tools of an agent's chats, Model.chatTokens), or that a chat left in the pool when a new request took its slot (vacate): its tokens (whole KV pages), the recurrent states after them, and their keys and values: on a GPU the KV pages of the pool that hold them, which the slots that start with them read in place (Engine.borrowed), on the host a copy per pool layer; used is the step a slot last took it. On a model in stages, stages holds each later stage's pages and states.

  • tokens: Vec<i64>
  • long: bool
  • saved: M70.Saved
  • pages: Vec<i64>
  • keys: Vec<Vec<f32>>
  • values: Vec<Vec<f32>>
  • used: i64
  • stages: Vec<M70.Shared>

Linear

gizlang
type Linear = { n: i64; k: i64; format: i64; codes: bytes; scales: bytes; book: Vec<f32>; turned: bool; }

turned: the linear reads its inputs unrotated in a rotated model (K formats), so x is turned back first.

  • n: i64
  • k: i64
  • format: i64
  • codes: bytes
  • scales: bytes
  • book: Vec<f32>
  • turned: bool

EngineLora

gizlang
type EngineLora = { names: Vec<string>; at: Vec<i64>; ranks: Vec<i64>; a: Vec<Buffer>; b: Vec<Buffer>; rows: Vec<Buffer>; low: Vec<Buffer>; slot: Vec<i64>; made: Vec<i64>; }

The LoRAs an engine serves on a GPU, many at once (Engine.adapt): names by index; for each device linear (by its index, DeviceLinear.id; at) its rank R (the most of any LoRA's; others padded with zeros) and the LoRAs' A [LoRAs, R, k] (each row rotated as the linear's inputs are) and B [LoRAs, n, R], scales in B, F32. Each step row's LoRA (-1: none) is in rows (I32), from each slot's (slot), each wanted row's after Engine.rows (the head's), and a linear's x A in low [rows, the largest R]. A slot's keys and values were made under made[slot]: another LoRA does not reuse them (admit).

  • names: Vec<string>
  • at: Vec<i64>
  • ranks: Vec<i64>
  • a: Vec<M7.Buffer>
  • b: Vec<M7.Buffer>
  • rows: Vec<M7.Buffer>
  • low: Vec<M7.Buffer>
  • slot: Vec<i64>
  • made: Vec<i64>

DeviceLinear

gizlang
type DeviceLinear = { n: i64; k: i64; format: i64; codes: Buffer; scales: Buffer; book: Buffer; turned: bool; maps: Vec<Buffer>; id: i64; }

On a GPU, a K format's book holds its subs' bytes (see K formats); maps are the copy engine's maps of an NV4 weight's codes and scales for gemmNvKernel (nvMaps), none for other formats.

  • n: i64
  • k: i64
  • format: i64
  • codes: M7.Buffer
  • scales: M7.Buffer
  • book: M7.Buffer
  • turned: bool
  • maps: Vec<M7.Buffer>
  • id: i64

Layer

gizlang
type Layer = { kind: i64; at: i64; wide: bool; source: i64; layerInput: Vec<i64>; window: i64; rope: i64; n1: i64; n2: i64; n3: i64; n4: i64; scalar: i64; qkv: i64; rest: i64; o: i64; qn: i64; kn: i64; bias: i64; input: i64; out: i64; conv: i64; decay: i64; dt: i64; gn: i64; up: i64; down: i64; ah: Vec<i64>; mh: Vec<i64>; router: i64; ple: Vec<i64>; }

A layer's tensors: indexes into the model's linears (qkv, o, input, out, up, down) and small tensors (the others), -1 where the layer has none. at is its index among the layers of its kind (an attention layer's: its pool layer, Engine.kv), wide whether its heads are Arch.wideAt's.

  • kind: i64
  • at: i64
  • wide: bool
  • source: i64
  • layerInput: Vec<i64>
  • window: i64
  • rope: i64
  • n1: i64
  • n2: i64
  • n3: i64
  • n4: i64
  • scalar: i64
  • qkv: i64
  • rest: i64
  • o: i64
  • qn: i64
  • kn: i64
  • bias: i64
  • input: i64
  • out: i64
  • conv: i64
  • decay: i64
  • dt: i64
  • gn: i64
  • up: i64
  • down: i64
  • ah: Vec<i64>
  • mh: Vec<i64>
  • router: i64
  • ple: Vec<i64>

Mtp

gizlang
type Mtp = { layer: Layer; fc: i64; ne: i64; nh: i64; norm: i64; }

The MTP layer (Qwen3.5, DeepSeek V3): a token's embedding and the main model's last hidden state at the position before it, each normed (ne, nh), joined by the fc linear, then one attention layer (its KV pages after the main attention layers') and its norm before the shared head: the most likely token after that token.

  • layer: M70.Layer
  • fc: i64
  • ne: i64
  • nh: i64
  • norm: i64

Drafter

gizlang
type Drafter = { shape: DraftShape; layers: Vec<Layer>; convs: Vec<Vec<i64>>; context: i64; fc: Vec<i64>; hn: i64; norm: i64; selector: i64; codebooks: Vec<bytes>; }

A block drafter on the model's GPU (DraftShape, draftParts), its tensors in Model.device and Model.small: per layer its Layer and its convolutions' base kernels and projections (before the attention, before the MLP: [ab, ap, mb, mp]; empty without them), the linear of every layer's context keys and values, the linear of each tap, the norms of the context and the output, and the selector's projection (-1: none) with its codebooks in host memory (BF16 rows of shape.rank: the token before, the candidate).

  • shape: _gizlang_lib_ai__llm__shape.DraftShape
  • layers: Vec<M70.Layer>
  • convs: Vec<Vec<i64>>
  • context: i64
  • fc: Vec<i64>
  • hn: i64
  • norm: i64
  • selector: i64
  • codebooks: Vec<bytes>

FlashFiles

gizlang
type FlashFiles = { fd: i64; at: Vec<i64>; record: i64; bits: i64; books: Vec<Buffer>; rows: Vec<Stored>; mult: Vec<u64>; sizes: Vec<i64>; offsets: Vec<i64>; }

A Flash model's own files: the experts' (its descriptor, the file offset of each layer's records, a record's bytes and its two codebooks on the GPU, gate and up, then down) and the PLE's n-gram rows (BF16; none without a PLE) with their hash multipliers, each head's vocabulary size and first row.

  • fd: i64
  • at: Vec<i64>
  • record: i64
  • bits: i64
  • books: Vec<M7.Buffer>
  • rows: Vec<gizlang_safetensors.Stored>
  • mult: Vec<u64>
  • sizes: Vec<i64>
  • offsets: Vec<i64>

Flash

gizlang
type Flash = { bufs: Vec<Buffer>; chunks: Ref<Vec<Buffer>>; perChunk: i64; most: i64; slotOf: Vec<i64>; owner: Vec<i64>; used: Vec<bool>; busy: Vec<i64>; hand: i64; stamp: i64; hits: i64; misses: i64; readNs: i64; history: Vec<i64>; convs: Vec<Vec<f32>>; last: Vec<Vec<i64>>; ahead: i64; }

Flash's state on a GPU (Engine.flash): its work buffers (F_*), the expert cache and the PLE's states. The cache holds an expert in each slot (expertBytes) in blocks (chunks) of perChunk slots: slots 0 to layers - 1 the layers' shared experts for good, the others the experts the routers picked last (CLOCK: a slot read since the hand passed it last stays once more). Per expert (layer (experts + 1) + index) its slot or -1; per slot its expert or -1, whether it was read and the stamp of the layer that reads it (not evicted then). The hits and misses of the cache and the ns its reads took. Per slot of the engine the last two tokens (the latest first) and the PLE convolution's last inputs.

  • bufs: Vec<M7.Buffer>
  • chunks: Ref<Vec<M7.Buffer>>
  • perChunk: i64
  • most: i64
  • slotOf: Vec<i64>
  • owner: Vec<i64>
  • used: Vec<bool>
  • busy: Vec<i64>
  • hand: i64
  • stamp: i64
  • hits: i64
  • misses: i64
  • readNs: i64
  • history: Vec<i64>
  • convs: Vec<Vec<f32>>
  • last: Vec<Vec<i64>>
  • ahead: i64

Thought

gizlang
type Thought = { open: string; close: string; }

How a model's reply marks its thinking: open is the text it writes to start a thought ("": the prompt opens it, as ChatML's <think>), close the text that ends it ("": the model does not think).

  • open: string
  • close: string

Chat

gizlang
type Chat = { template: Template; names: Vec<string>; values: Vec<Value>; xhigh: bool; }

A model's chat template (chat_template.jinja or tokenizer_config.json's chat_template) as Hugging Face renders it, the special tokens it may name (names, their values: bos_token, eos_token), and whether its top reasoning effort is called xhigh (Qwen 3.8), where a chat's "high" goes.

  • template: gizlang_jinja.Template
  • names: Vec<string>
  • values: Vec<gizlang_jinja.Value>
  • xhigh: bool

Model

gizlang
type Model = { arch: Arch; folder: string; tokenizer: Tokenizer; stops: Vec<i64>; chat?: Chat; thought: Thought; call: string; sampling: Sampling; embed: Stored; layers: Vec<Layer>; norm: i64; head: i64; mtp: Option<Mtp>; mixer: Vec<i64>; drafter?: Drafter; linears: Vec<Linear>; small: Vec<Vec<f32>>; gpu: Option<Gpu>; device: Vec<DeviceLinear>; deviceSmall: Vec<Buffer>; signs: Vec<Buffer>; sources: Vec<Vec<string>>; embedTable?: Mapped; layerInput?: { embed: Stored; projection: i64; norm: i64; }; files: FlashFiles; vision?: Vision; weights?: Stack; spots: Vec<Vec<Spot>>; beside: i64; stages: Vec<Model>; kvCosts?: string; npu?: NpuLlm.Model; }

A converted model, on the GPU that gizhw/gpu opens (GIZLANG_DEVICE) when there is one (device), else on the host (linears); the small tensors are on both. The embedding stays in the file, and where the GPU of the first layer reads it (embedTable, Gpu.mapFile): a step there gathers its tokens' rows on the GPU, the host reads them from the file. chat its chat template when it has one (Chat), thought how its reply marks its thinking (Thought), call the mark where its tool calls start (gizai/llm/model CALL, PYTHON_CALL; "": it writes none); stops are the tokens that end a generation and sampling the model's own settings. mtp is the MTP layer on a GPU when the model has one, vision its vision tower when its folder has one (convertVision), stages the models of its later layers on other GPUs (loadAcross).

  • arch: _gizlang_lib_ai__llm__shape.Arch
  • folder: string
  • tokenizer: M26.Tokenizer
  • stops: Vec<i64>
  • chat: _gizlang_sdk_core.Option<M70.Chat>
  • thought: M70.Thought
  • call: string
  • sampling: M72.Sampling
  • embed: gizlang_safetensors.Stored
  • layers: Vec<M70.Layer>
  • norm: i64
  • head: i64
  • mtp: _gizlang_sdk_core.Option<M70.Mtp>
  • mixer: Vec<i64>
  • drafter: _gizlang_sdk_core.Option<M70.Drafter>
  • linears: Vec<M70.Linear>
  • small: Vec<Vec<f32>>
  • gpu: _gizlang_sdk_core.Option<M7.Gpu>
  • device: Vec<M70.DeviceLinear>
  • deviceSmall: Vec<M7.Buffer>
  • signs: Vec<M7.Buffer>
  • sources: Vec<Vec<string>>
  • embedTable: _gizlang_sdk_core.Option<M7.Mapped>
  • layerInput: _gizlang_sdk_core.Option<{ embed: gizlang_safetensors.Stored; projection: i64; norm: i64; }>
  • files: M70.FlashFiles
  • vision: _gizlang_sdk_core.Option<_gizlang_lib_ai__llm__vision.Vision>
  • weights: _gizlang_sdk_core.Option<_gizlang_lib_ai__parts__stack.Stack>
  • spots: Vec<Vec<M70.Spot>>
  • beside: i64
  • stages: Vec<M70.Model>
  • kvCosts: _gizlang_sdk_core.Option<string>
  • npu: _gizlang_sdk_core.Option<_gizlang_lib_ai__llm__npu__gemma.Model>

Spot

gizlang
type Spot = { linear: i64; codes: i64; scales: i64; book: i64; }

A linear of a layer's group (Model.weights): its place among the model's device linears, and which of the group's tensors are its codes, its scales and its book.

  • linear: i64
  • codes: i64
  • scales: i64
  • book: i64

Functions

deltaTerms

gizlang
function deltaTerms(a: Arch): i64

Floats of a draft row's delta rule terms per value head: its normed key, value, decay and beta.

Effects: fail.

descRows

gizlang
function descRows(rows: i64): i64

The descriptor of a step (W_DESC, f32) for rows rows: per row its slot and position, per sequence its slot, first row, row count and drafts (its last rows, see Engine.generate), per row the rotary cosines and sines, the wanted rows, then the tiles (first row and rows, see tileKernel), the local rotary table and the rows' tokens (embedKernel).

descSegments

gizlang
function descSegments(rows: i64): i64

Where the per-sequence part of the descriptor starts.

Effects: fail.

descRope

gizlang
function descRope(rows: i64): i64

Where the rows' rotary cosines and sines start.

Effects: fail.

descWanted

gizlang
function descWanted(a: Arch, rows: i64): i64

Where the wanted rows start.

Effects: fail.

descTiles

gizlang
function descTiles(a: Arch, rows: i64): i64

Where the tiles start.

Effects: fail.

descLocal

gizlang
function descLocal(a: Arch, rows: i64): i64

The local layers' rotary table (Arch.local), after the tiles.

Effects: fail.

descTokens

gizlang
function descTokens(a: Arch, rows: i64): i64

Where the rows' tokens start (embedKernel).

Effects: fail.

descSize

gizlang
function descSize(a: Arch, rows: i64): i64

The floats of the whole descriptor.

Effects: fail.

addressAt

gizlang
function addressAt(b: Buffer, floats: i64): u64

The address FLOATS floats into B.

Effects: fail.

partCapacity

gizlang
function partCapacity(rows: i64): i64

The parts the parts buffer holds for an engine of rows rows.

Effects: fail.

partFloats

gizlang
function partFloats(headDim: i64): i64

The floats of a split's partial result of one row and query head (attention's parts, the drafter's too): its weighted values (headDim), their largest score and the sum of their weights.

Effects: fail.

stretches

gizlang
function stretches(e: Engine, prompt: i64, limit: i64): bool

Whether a request of prompt tokens that writes at most limit is long: the model has a long table (Arch.long) and the request may pass Arch.longFrom. The others turn by the plain table (logits as within the trained context); one that passes longFrom all the same goes on with it.

Effects: fail.

attentionLayers

gizlang
function attentionLayers(a: Arch): i64

The attention layers of a.

Effects: fail.

recurrentLayers

gizlang
function recurrentLayers(a: Arch): i64

The recurrent (non-attention) layers of a.

Effects: fail.

drafting

gizlang
function drafting(e: Engine): bool

Whether generate drafts with the model's MTP layer (on a model in stages, the last stage's: headOf).

Effects: fail.

looksUp

gizlang
function looksUp(e: Engine): bool

Whether generate drafts by lookup (a model without an MTP layer or block drafter; not on an NPU, which reads a token a step).

Effects: fail.

mtpOn

gizlang
function mtpOn(e: Engine): bool

Whether this engine's own model runs the MTP layer: its buffers and KV pages hold it.

Effects: fail.

poolLayers

gizlang
function poolLayers(e: Engine): i64

Attention layers in the KV pages: the model's, and its MTP layer's when generate drafts with it.

Effects: fail.

pageBytes

gizlang
function pageBytes(e: Engine): i64

The bytes of a KV page: keys and values of PAGE positions in each pool layer (Engine.kv).

poolWides

gizlang
function poolWides(e: Engine): Vec<bool>

Per pool layer (Layer.at): whether its heads are the wide shape (Arch.wideView); the MTP layer's last.

Effects: fail, alloc.

kvGroup

gizlang
function kvGroup(e: Engine, at: i64): KvGroup

The group of the KV pages that keeps the keys and values pool layer at (Layer.source) reads.

Effects: fail.

hostWidth

gizlang
function hostWidth(e: Engine, l: i64): i64

Floats of a position's keys (or values) in pool layer l (Layer.at) on the host.

Effects: fail.

widestInput

gizlang
function widestInput(a: Arch): i64

Floats of the widest input of a linear and of the widest output.

Effects: fail.

widestOutput

gizlang
function widestOutput(a: Arch): i64

Floats of the widest output of a linear, rounded up to 16.

Effects: fail, alloc.

attentionWidth

gizlang
function attentionWidth(a: Arch): i64

Floats of the widest attention output of a row: heads times the widest head.

Effects: fail.

wantedRows

gizlang
function wantedRows(e: Engine): i64

The most rows of a step whose logits a GPU engine gives: 1 + LEVELS a sequence, and the recalled drafts of one sequence beyond those (Engine.advance recalls for one a step): 1 + depth a sequence would hold 0.5 GB of logits at 8 slots.

Effects: fail.

draftRows

gizlang
function draftRows(e: Engine): i64

The draft rows each recurrent layer of W_DRAFT_CONV and W_DRAFT_DELTA holds (draftBase): a step's drafts are among its wanted rows.

Effects: fail.

headOf

gizlang
function headOf(e: Engine): Engine

The engine whose model has the head: the last stage's (Engine.stages), else e.

Effects: fail.

noTurn

gizlang
function noTurn(): Turn

A turn of no request: the turn of an empty slot.

Effects: fail, alloc.

noSaved

gizlang
function noSaved(): Saved

No saved states.

noLora

gizlang
function noLora(slots: i64): EngineLora

An engine of slots slots that serves no LoRA.

Effects: fail, alloc.

noFiles

gizlang
function noFiles(): FlashFiles

FlashFiles of no file.

Effects: fail.

noFlash

gizlang
function noFlash(): Flash

The Flash state of an engine without Flash.

Effects: fail, write, alloc.

noThought

gizlang
function noThought(): Thought

The Thought of a model that does not think.

interleaved

gizlang
function interleaved(states: Vec<Vec<f32>>, rows: i64, dim: i64): Vec<f32>

Per row the rows of each of states (rows of dim floats each), in turn.

Effects: fail, alloc.

embedded

gizlang
function embedded(e: Engine, tokens: Vec<i64>): Vec<f32> throws string

The embedding rows of the tokens, read from the model file.

Effects: fail, read, write, console, io, alloc.

ropeTable

gizlang
function ropeTable(a: Arch, positions: Vec<i64>, grid: Vec<i64> = [], table: i64 = GLOBAL, long: Vec<bool> = []): Vec<f32>

The rotary cosines and sines of each row's position in rotary table table (GLOBAL, LOCAL): rotary/2 pairs a row, on the global table of the long frequencies times Arch.longScale for a row of a long sequence (long, Rows.long), else times Arch.ropeScale. With a grid (three positions a row: time, height, width) frequency j takes height's where j % 3 is 1 and width's where it is 2, within 3 times their sections (a.sections), time's elsewhere: Qwen3-VL's interleaved multimodal rotary positions.

Effects: fail, alloc.

pleKept

gizlang
function pleKept(): i64

Positions of the PLE convolution's inputs a slot keeps: (taps - 1) times the dilation.

Effects: fail.

blocksOn

gizlang
function blocksOn(e: Engine): bool

Whether this engine drafts with its model's block drafter (Model.drafter): on one GPU.

ringPositions

gizlang
function ringPositions(d: DraftShape, depth: i64): i64

The positions a slot's ring holds for each drafter layer on an engine of depth drafts: the window, and the most rows a step writes past a sequence's next token (a block's later rows, or recalled drafts), so the rows a step rejects land where the window no longer reads.

Effects: fail.

Methods

Engine.reset

gizlang
function Engine.reset(self: mut Engine, slot: i64): void

Empties a slot for a new sequence.

Effects: fail, read, write, console, io, alloc.

Constants

W_X

gizlang
const W_X: i64 = 0

The rows' hidden states (rows by dim).

W_Y

gizlang
const W_Y: i64 = 1

A product's input, normed (rows by the widest input).

W_P

gizlang
const W_P: i64 = 2

A product's output (rows by the widest output).

W_Q

gizlang
const W_Q: i64 = 3

The rows' queries and attention's output (rows by the attention width).

W_C

gizlang
const W_C: i64 = 4

The convolution's inputs and outputs (rows by convDim).

W_PARTS

gizlang
const W_PARTS: i64 = 5

Attention's partial results of each split (partCapacity).

W_DESC

gizlang
const W_DESC: i64 = 6

The step's descriptor (descSize).

W_MAP

gizlang
const W_MAP: i64 = 7

The slot of each row.

W_TABLE

gizlang
const W_TABLE: i64 = 8

The page table: each slot's KV pages.

W_POOL

gizlang
const W_POOL: i64 = 9

The KV pool: the pages of every slot's keys and values.

W_CONV

gizlang
const W_CONV: i64 = 10

Each slot's last convolution inputs.

W_STATE

gizlang
const W_STATE: i64 = 11

Each slot's delta rule states, BF16 two to a float.

W_SCRATCH

gizlang
const W_SCRATCH: i64 = 12

The partial sums of a split product, or the wanted rows' logits.

W_HID

gizlang
const W_HID: i64 = 13

The last hidden states of a step's rows, normed, for the MTP layer.

W_EMB

gizlang
const W_EMB: i64 = 14

The embeddings of the MTP layer's tokens.

W_SRC

gizlang
const W_SRC: i64 = 15

The step rows whose hidden states the MTP layer reads.

W_LOGITS

gizlang
const W_LOGITS: i64 = 16

The wanted rows' logits.

W_TOP

gizlang
const W_TOP: i64 = 17

The wanted rows' most likely tokens.

W_DRAFT_CONV

gizlang
const W_DRAFT_CONV: i64 = 18

Each slot's convolution inputs of its drafts in the last step (Engine.generate).

W_DRAFT_DELTA

gizlang
const W_DRAFT_DELTA: i64 = 19

Each slot's delta rule terms of its drafts in the last step (deltaTerms).

W_DRAFT_BLOCKS

gizlang
const W_DRAFT_BLOCKS: i64 = 20

The head blocks a draft reads (Engine.draftBlocks).

W_CAND

gizlang
const W_CAND: i64 = 21

The wanted rows' sampling candidates (candidatesKernel).

W_X8

gizlang
const W_X8: i64 = 22

x of an FP8 product (fp8Kernel): its rows' scales (big floats), then its E4M3 values, 4 to a float; or x turned back from its rotation for a K format (plainKernel).

W_LAYER_INPUT

gizlang
const W_LAYER_INPUT: i64 = 23

The rows' per-layer inputs (rows by layers by layerInput), after their projection.

W_LAYER_EMBED

gizlang
const W_LAYER_EMBED: i64 = 24

The rows' per-layer embeddings (rows by layers by layerInput), as the host looked them up.

SPLITS

gizlang
const SPLITS: i64 = 16

Splits of the keys that attention reads at once: a step by tiles takes partSplits; attendKernel (fewer rows) more, up to two waves of its workgroups on the GPU (Gpu.slots; ATTEND_GROUPS where the driver does not say) within the parts buffer (partCapacity), each split of at least SPAN keys. (One row's 4 kv heads in 16 splits kept 64 workgroups on 76 multiprocessors: 60K keys took 0.8 ms a layer, nine times what the memory takes to read them.)

SPAN

gizlang
const SPAN: i64 = 64

The fewest keys of a split.

ATTEND_GROUPS

gizlang
const ATTEND_GROUPS: i64 = 512

The workgroups of attention's two waves where the driver does not say (Gpu.slots).

LOG2E

gizlang
const LOG2E = Math.LOG2E

log2 e: softmax weights are 2^(x log2 e) on the GPU's exp2 (its exp polynomial cost some 25 instructions a weight).

GEMV_ROWS

gizlang
const GEMV_ROWS: i64 = 8

The most rows a GPU linear reads its weights for at once on CUDA cores; from GEMM_ROWS rows a linear runs on matrix cores where the GPU has them.

GEMM_ROWS

gizlang
const GEMM_ROWS: i64 = 32

The fewest rows a linear runs on matrix cores for.

PACE_CLASSES

gizlang
const PACE_CLASSES: i64 = 4

The classes of context length Drafting.steps keeps apart (paceClass).

DRAFTS

gizlang
const DRAFTS: i64 = 32

The drafts an engine reads per sequence and step by default: recalled drafts (LlmGenerate.recalled) take all of them; those of the MTP layer or lookup at most LEVELS, as the n-th is kept less often than the one before and each costs a row (and with an MTP layer a step of it and the head).

LEVELS

gizlang
const LEVELS: i64 = 8

The most drafts an MTP layer or lookup makes a step (LlmGenerate.draftLevels).

DRAFT_SHARE

gizlang
const DRAFT_SHARE: i64 = 8

A draft reads one DRAFT_SHARE-th of the head's blocks of 16 rows (as FR-Spec, arXiv 2502.14856, which takes the frequent tokens of a corpus): the first half of them always (a BPE vocabulary's first ids are its frequent tokens), the other half the last blocks of the tokens the engine read, so the language of a prompt comes in with it.

MTP_WINDOW

gizlang
const MTP_WINDOW: i64 = 32768

The last keys the MTP layer attends to: its drafts lean on the near text, and the model checks every one, so the output is the same; at 252K a draft pass reads an eighth of the keys.

RECORDINGS

gizlang
const RECORDINGS: i64 = 64

The most step shapes an engine keeps recorded.

RECORD_ROOM

gizlang
const RECORD_ROOM: i64 = 50_331_648

The device memory free before a new recording at least (one of q27-4's steps took 6 MiB).

PLE_TAPS

gizlang
const PLE_TAPS: i64 = 4

The taps of the PLE convolution.

PLE_DILATION

gizlang
const PLE_DILATION: i64 = 3

The dilation of the PLE convolution.

SPLIT_ROOM

gizlang
const SPLIT_ROOM: i64 = 1_073_741_824

The device memory a model's weights leave free on its GPU at least, for an engine's work buffers and KV pages: a model that leaves less takes the GPU's other GPUs too (loadAcross), or keeps layers in host memory (Model.weights).