gizai/llm/state
the data of a model and of its engine: their records, the constants of their shapes, the pure reads over them and the reset of a slot. The modules that run an engine (kv, gemm, delta, gpu, host, flash, generate, engine) and the module that loads a model (model) import it. It imports none of them.
import * as State from "gizai/llm/state";Types
Engine
type Engine = { model: Model; slots: i64; context: i64; rows: i64; lengths: Vec<i64>; long: Vec<bool>; kv: KvPlan; host: HostState; device: DeviceState; draft: Drafting; queue: Queue; prefix: PrefixCache; vocabulary: Vocabulary; flash: Flash; stages: Vec<Engine>; exact: bool; fuse: Vec<string>; lora: EngineLora; block?: BlockState; }The engine of a model: slots sequences of at most context positions, steps of at most rows rows.
model:M70.Modelslots:i64context:i64rows:i64lengths:Vec<i64>long:Vec<bool>kv:M65.KvPlanhost:M70.HostStatedevice:M70.DeviceStatedraft:M70.Draftingqueue:M70.Queueprefix:M70.PrefixCachevocabulary:_gizlang_lib_ai__llm__grammar.Vocabulary— The tokens' bytes for grammars (Tokenizer.tokenBytes, sorted), read at the first request with a grammar.flash:M70.Flashstages:Vec<M70.Engine>exact:bool— Linears give x to the GEMMs as the F32 the model computes (rounded to F16 there): no gemm16Kernel, whose 16-bit weights take x in their own type (BF16 for BF16 weights, as other servers' activations), and NV4 and F8 weights run as GEMVs, not GEMMs of x in NVFP4 or FP8. Off: the faster.fuse:Vec<string>— The fusions the engine takes (EngineOptions.fuse, FUSIONS):conv, the delta rule's convolution within its chunks' terms (deltaPrepKernel), andnv, gate_up's activations as the down GEMM's NVFP4 input (gemmNvKernel'snv). Neither was faster on the RTX 5090 at 2048 rows, so both are off by default.lora:M70.EngineLora— The LoRAs it serves (Engine.adapt): a request names one (Request.adapter).block:_gizlang_sdk_core.Option<M70.BlockState>
BlockState
type BlockState = { ring: Buffer; rope: Buffer; meta: Buffer; parts: Buffer; mixed: Buffer; from: Vec<i64>; high: Vec<i64>; }A block drafter's state (Drafter): per slot its context keys and values of each drafter layer at the last positions
(ring, F16: slot, layer, position modulo ringPositions, then the kv heads' keys and their values), the rotary table of
the rows of a step (rope), a draft pass's tokens, sequences and rows (meta, blockMeta), its attention's partial
results (parts) and the inputs of its linears (mixed). Per slot the first position its ring holds for the sequence
there (from) and the end of the positions written to it (high): a draft reads positions from both on.
ring:M7.Bufferrope:M7.Buffermeta:M7.Bufferparts:M7.Buffermixed:M7.Bufferfrom:Vec<i64>high:Vec<i64>
HostState
type HostState = { keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; convs: Vec<Vec<f32>>; states: Vec<Vec<f32>>; replay: Vec<Vec<f32>>; hidden: Vec<f32>; }An engine's state on the host: per slot and pool layer the keys and values (position, kv head, dimension; as kvStore keeps them, see kvRounded), per slot and recurrent layer the convolution's last inputs and the delta rule's state, and its last step's drafts' convolution inputs and delta rule terms (cpuReplay); the last hidden states that the MTP layer reads.
keys:Vec<Vec<f32>>values:Vec<Vec<f32>>convs:Vec<Vec<f32>>states:Vec<Vec<f32>>replay:Vec<Vec<f32>>hidden:Vec<f32>
DeviceState
type DeviceState = { work: Vec<Buffer>; pages: Vec<Vec<i64>>; table: Vec<f32>; free: Vec<i64>; maxPages: i64; seen: Vec<string>; shapes: Vec<string>; recordings: Vec<G.Recording>; seeing: i64; slack: i64; placed: i64; held: Ref<Vec<Buffer>>; taps: Ref<Vec<Buffer>>; deep: Ref<Vec<Buffer>>; }An engine's state on a GPU: the work buffers (W_*), each slot's pages, the page table and the free pages; the step
shapes launched once, the recordings of those launched again, the bytes a picture's pass takes (Vision.peak), those
the driver may keep after passes (slack) and the buffer of both, held between passes (pictureRoom; a cell: the
copies of a served engine share it, and a copy that kept its own Vec kept the buffer, 11/6).
work:Vec<M7.Buffer>pages:Vec<Vec<i64>>table:Vec<f32>free:Vec<i64>maxPages:i64seen:Vec<string>shapes:Vec<string>recordings:Vec<M7.Recording>seeing:i64slack:i64placed:i64held:Ref<Vec<M7.Buffer>>taps:Ref<Vec<M7.Buffer>>— The buffer of the hidden states a step keeps (Engine.hidden), of the engine's largest step: made at the first such step and held while the engine lives, so an encoder's prompts ask the card for no other.deep:Ref<Vec<M7.Buffer>>— The buffers of a picture's rows for the deeper layers in such a step (Visual.deep), each of the largest step: held astaps.
Drafting
type Drafting = { depth: i64; pending: Vec<i64>; bases: Vec<i64>; drafts: i64; kept: i64; draftBlocks: Vec<i64>; blockEntry: Vec<i64>; nextEntry: i64; steps: Vec<f64>; reach: Vec<f64>; ahead: Vec<Vec<Ahead>>; level: i64; forced: i64; lookup: i64; recall: i64; }Drafts and pacing: the most drafts a sequence's step reads (0: none; see Engine.step), per slot its drafts of the last step not accepted yet and their first draft row (draftBase), and the drafts generate read and kept. The head blocks a draft reads (DRAFT_SHARE; empty: all; on a GPU also in W_DRAFT_BLOCKS), per head block its entry there or -1, and the entry of the second half the next new block takes. Per context class c (paceClass), count n of generating sequences and count i of draftLevels(depth), the ns a step took (a moving average; 0: not measured yet) at steps[(c * (slots + 1) + n) * levels + i]. Per depth i, the share of drafts right through draft i (reach[i], a moving average; -1: none judged yet), and per slot its drafts not judged yet (Ahead). The drafts of the last step and the count warming forces (-1: none; draftLevel). The tokens a lookup and a recalled draft match before them (EngineOptions.lookup, EngineOptions.recall).
depth:i64pending:Vec<i64>bases:Vec<i64>drafts:i64kept:i64draftBlocks:Vec<i64>blockEntry:Vec<i64>nextEntry:i64steps:Vec<f64>reach:Vec<f64>ahead:Vec<Vec<M70.Ahead>>level:i64forced:i64lookup:i64recall:i64
Ahead
type Ahead = { at: i64; tokens: Vec<i64>; }Drafts a source made after a sequence's next token, the first at index at of its held tokens: all it made, also
those the step did not read, judged against the tokens the sequence holds later (LlmGenerate.judged).
at:i64tokens:Vec<i64>
Queue
type Queue = { waiting: Vec<Turn>; turns: Vec<Turn>; next: Vec<i64>; guesses: Vec<Vec<Guess>>; held: Vec<Vec<i64>>; freed: Vec<i64>; saved: Vec<Saved>; grams: Vec<Map<i64, i64>>; chains: Vec<Vec<i64>>; ids: i64; steps: i64; made: i64; read: i64; }The requests in turn (Engine.add): those waiting for a slot, and per slot the one it serves (id -1: none), the token it reads next (-1: none yet, it reads its prompt) and its drafts of the tokens after that. Per slot the tokens its keys, values and states hold (a free slot keeps them for a prompt that starts with them), the step it was last freed at, the recurrent states it saved at a prompt's mark, and the index of its tokens' pairs (Queue.grams: the place after the latest of each pair; chains[i]: the place after the pair before place i, -1: none; recalled). The next request id and step.
waiting:Vec<M70.Turn>turns:Vec<M70.Turn>next:Vec<i64>guesses:Vec<Vec<M70.Guess>>held:Vec<Vec<i64>>freed:Vec<i64>saved:Vec<M70.Saved>grams:Vec<_gizlang_sdk_collections.Map<i64, i64>>chains:Vec<Vec<i64>>ids:i64steps:i64made:i64read:i64
PrefixCache
type PrefixCache = { shared: Vec<Shared>; borrowed: Vec<i64>; stored: Vec<Prefix>; reuse: bool; folder: string; }The prefix cache: the starts many prompts share (Shared), per slot how many of its first KV pages are a shared
start's, and the starts of chats that left their slots, on the host or, past the host memory free, in files of
folder (see Prefix; "?" until first used, "" for none). With reuse off each prompt reads all of its tokens.
shared:Vec<M70.Shared>borrowed:Vec<i64>stored:Vec<M70.Prefix>reuse:boolfolder:string
Rows
type Rows = { tokens: Vec<i64>; slots: Vec<i64>; positions: Vec<i64>; ropes: Vec<i64>; grid: Vec<i64>; own: i64; wanted: Vec<i64>; segments: Vec<i64>; drafts: Vec<i64>; layerInputs?: Vec<f32>; }A step: per row its token, slot, position and rotary position (or three: grid, Visual.grid), the rows whose logits are wanted, per sequence its
slot, first row and row count, and per sequence its drafts (its last rows, see Engine.generate). A row sees the keys
before its position, and its own when own is 1 (0: a padding row, see Engine.padding).
tokens:Vec<i64>slots:Vec<i64>positions:Vec<i64>ropes:Vec<i64>grid:Vec<i64>own:i64wanted:Vec<i64>segments:Vec<i64>drafts:Vec<i64>layerInputs:_gizlang_sdk_core.Option<Vec<f32>>
ExpertCache
type ExpertCache = { slots: i64; hits: i64; misses: i64; readMs: f64; }The expert cache of an engine: its slots, the experts found in it and read into it, and the ms the reads took.
slots:i64hits:i64misses:i64readMs:f64
Guess
type Guess = { token: i64; q: Dist; }A draft per sequence: its token and the distribution it was drawn from (all on the token when it is the most likely).
token:i64q:M72.Dist
Progress
type Progress = { id: i64; tokens: Vec<i64>; cached: i64; end: string; logprobs: Vec<Logprob>; }What a step gave a request (Engine.advance): its new tokens (without a stop token), the tokens at the start of its prompt that its slot held already (not read again), and why it ended: "stop" (a stop token), "length" (its limit, the context's end, or a KV cache too small for it alone) or "" (it goes on).
id:i64tokens:Vec<i64>cached:i64end:stringlogprobs:Vec<M72.Logprob>
Turn
type Turn = { id: i64; request: Request; random: Vec<u64>; made: i64; mark: i64; head: i64; cached: i64; long: bool; since: i64; reading?: Recognizer; }A request in an engine's turn: its id, its random state, the tokens it gave, the prompt position after which its slot saves the recurrent states (Engine.add), the position after which its prompt's start is shared (Engine.add; 0: none), the prompt tokens its slot held already, its grammar's recognizer while the grammar holds (Sampling), whether its sequence is long (stretches), and the ns (Task.nanos) it began to wait, or to run once it took a slot (admit).
id:i64request:M72.Requestrandom:Vec<u64>made:i64mark:i64head:i64cached:i64long:boolsince:i64reading:_gizlang_sdk_core.Option<_gizlang_lib_ai__llm__grammar.Recognizer>
Saved
type Saved = { at: i64; conv: Vec<f32>; state: Vec<f32>; pages: Vec<i64>; }A slot's recurrent states after its first at tokens (0: none): the convolutions' last inputs and the delta rules'
states. On a GPU they are in KV pages of the pool (pages, savedPages: a copy on the device, not over the bus), or
on the host where the pool had no room or the start went to the host (Prefix); on the host conv and state (from
a GPU the states' BF16, two to a float). One holder owns the pages: a copy moves them or copies them on the device.
at:i64conv:Vec<f32>state:Vec<f32>pages:Vec<i64>
Prefix
type Prefix = { tokens: Vec<i64>; long: bool; saved: Saved; pages: Vec<bytes>; keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; used: i64; stages: Vec<Prefix>; file: string; ple: Vec<f32>; }The start of a chat that left its slot for another (see admit): its tokens, the recurrent states after them, and
their keys and values (on a GPU the bytes of their KV pages, on the host per pool layer); used is the step it was
stored at. A later prompt that starts with the tokens takes them back into a slot and reads only the rest. On a
model in stages, stages holds each later stage's pages and states (Engine.stages). A long sequence's start (Turn.long)
is taken back by a long one only: their keys turn by another table.
tokens:Vec<i64>long:boolsaved:M70.Savedpages:Vec<bytes>keys:Vec<Vec<f32>>values:Vec<Vec<f32>>used:i64stages:Vec<M70.Prefix>file:stringple:Vec<f32>
Shared
type Shared = { tokens: Vec<i64>; long: bool; saved: Saved; pages: Vec<i64>; keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; used: i64; stages: Vec<Shared>; }A start that many prompts share (as the system turn and tools of an agent's chats, Model.chatTokens), or that a chat
left in the pool when a new request took its slot (vacate): its tokens (whole KV pages), the recurrent states after
them, and their keys and values: on a GPU the KV pages of the pool that hold them, which the slots that start with
them read in place (Engine.borrowed), on the host a copy per pool layer; used is the step a slot last took it. On
a model in stages, stages holds each later stage's pages and states.
tokens:Vec<i64>long:boolsaved:M70.Savedpages:Vec<i64>keys:Vec<Vec<f32>>values:Vec<Vec<f32>>used:i64stages:Vec<M70.Shared>
Linear
type Linear = { n: i64; k: i64; format: i64; codes: bytes; scales: bytes; book: Vec<f32>; turned: bool; }turned: the linear reads its inputs unrotated in a rotated model (K formats), so x is turned back first.
n:i64k:i64format:i64codes:bytesscales:bytesbook:Vec<f32>turned:bool
EngineLora
type EngineLora = { names: Vec<string>; at: Vec<i64>; ranks: Vec<i64>; a: Vec<Buffer>; b: Vec<Buffer>; rows: Vec<Buffer>; low: Vec<Buffer>; slot: Vec<i64>; made: Vec<i64>; }The LoRAs an engine serves on a GPU, many at once (Engine.adapt): names by index; for each device linear (by its
index, DeviceLinear.id; at) its rank R (the most of any LoRA's; others padded with zeros) and the LoRAs' A [LoRAs, R, k]
(each row rotated as the linear's inputs are) and B [LoRAs, n, R], scales in B, F32. Each step row's LoRA (-1: none)
is in rows (I32), from each slot's (slot), each wanted row's after Engine.rows (the head's), and a linear's x A in
low [rows, the largest R]. A slot's keys and
values were made under made[slot]: another LoRA does not reuse them (admit).
names:Vec<string>at:Vec<i64>ranks:Vec<i64>a:Vec<M7.Buffer>b:Vec<M7.Buffer>rows:Vec<M7.Buffer>low:Vec<M7.Buffer>slot:Vec<i64>made:Vec<i64>
DeviceLinear
type DeviceLinear = { n: i64; k: i64; format: i64; codes: Buffer; scales: Buffer; book: Buffer; turned: bool; maps: Vec<Buffer>; id: i64; }On a GPU, a K format's book holds its subs' bytes (see K formats); maps are the copy engine's maps of an
NV4 weight's codes and scales for gemmNvKernel (nvMaps), none for other formats.
n:i64k:i64format:i64codes:M7.Bufferscales:M7.Bufferbook:M7.Bufferturned:boolmaps:Vec<M7.Buffer>id:i64
Layer
type Layer = { kind: i64; at: i64; wide: bool; source: i64; layerInput: Vec<i64>; window: i64; rope: i64; n1: i64; n2: i64; n3: i64; n4: i64; scalar: i64; qkv: i64; rest: i64; o: i64; qn: i64; kn: i64; bias: i64; input: i64; out: i64; conv: i64; decay: i64; dt: i64; gn: i64; up: i64; down: i64; ah: Vec<i64>; mh: Vec<i64>; router: i64; ple: Vec<i64>; }A layer's tensors: indexes into the model's linears (qkv, o, input, out, up, down) and small tensors (the
others), -1 where the layer has none. at is its index among the layers of its kind (an attention layer's: its pool
layer, Engine.kv), wide whether its heads are Arch.wideAt's.
kind:i64at:i64wide:boolsource:i64layerInput:Vec<i64>window:i64rope:i64n1:i64n2:i64n3:i64n4:i64scalar:i64qkv:i64rest:i64o:i64qn:i64kn:i64bias:i64input:i64out:i64conv:i64decay:i64dt:i64gn:i64up:i64down:i64ah:Vec<i64>mh:Vec<i64>router:i64ple:Vec<i64>
Mtp
type Mtp = { layer: Layer; fc: i64; ne: i64; nh: i64; norm: i64; }The MTP layer (Qwen3.5, DeepSeek V3): a token's embedding and the main model's last hidden state at the position before it, each normed (ne, nh), joined by the fc linear, then one attention layer (its KV pages after the main attention layers') and its norm before the shared head: the most likely token after that token.
layer:M70.Layerfc:i64ne:i64nh:i64norm:i64
Drafter
type Drafter = { shape: DraftShape; layers: Vec<Layer>; convs: Vec<Vec<i64>>; context: i64; fc: Vec<i64>; hn: i64; norm: i64; selector: i64; codebooks: Vec<bytes>; }A block drafter on the model's GPU (DraftShape, draftParts), its tensors in Model.device and Model.small: per layer its Layer and its convolutions' base kernels and projections (before the attention, before the MLP: [ab, ap, mb, mp]; empty without them), the linear of every layer's context keys and values, the linear of each tap, the norms of the context and the output, and the selector's projection (-1: none) with its codebooks in host memory (BF16 rows of shape.rank: the token before, the candidate).
shape:_gizlang_lib_ai__llm__shape.DraftShapelayers:Vec<M70.Layer>convs:Vec<Vec<i64>>context:i64fc:Vec<i64>hn:i64norm:i64selector:i64codebooks:Vec<bytes>
FlashFiles
type FlashFiles = { fd: i64; at: Vec<i64>; record: i64; bits: i64; books: Vec<Buffer>; rows: Vec<Stored>; mult: Vec<u64>; sizes: Vec<i64>; offsets: Vec<i64>; }A Flash model's own files: the experts' (its descriptor, the file offset of each layer's records, a record's bytes and its two codebooks on the GPU, gate and up, then down) and the PLE's n-gram rows (BF16; none without a PLE) with their hash multipliers, each head's vocabulary size and first row.
fd:i64at:Vec<i64>record:i64bits:i64books:Vec<M7.Buffer>rows:Vec<gizlang_safetensors.Stored>mult:Vec<u64>sizes:Vec<i64>offsets:Vec<i64>
Flash
type Flash = { bufs: Vec<Buffer>; chunks: Ref<Vec<Buffer>>; perChunk: i64; most: i64; slotOf: Vec<i64>; owner: Vec<i64>; used: Vec<bool>; busy: Vec<i64>; hand: i64; stamp: i64; hits: i64; misses: i64; readNs: i64; history: Vec<i64>; convs: Vec<Vec<f32>>; last: Vec<Vec<i64>>; ahead: i64; }Flash's state on a GPU (Engine.flash): its work buffers (F_*), the expert cache and the PLE's states. The cache
holds an expert in each slot (expertBytes) in blocks (chunks) of perChunk slots: slots 0 to layers - 1 the
layers' shared experts for good, the others the experts the routers picked last (CLOCK: a slot read since the hand
passed it last stays once more). Per expert (layer (experts + 1) + index) its slot or -1; per slot its expert or -1,
whether it was read and the stamp of the layer that reads it (not evicted then). The hits and misses of the cache
and the ns its reads took. Per slot of the engine the last two tokens (the latest first) and the PLE convolution's
last inputs.
bufs:Vec<M7.Buffer>chunks:Ref<Vec<M7.Buffer>>perChunk:i64most:i64slotOf:Vec<i64>owner:Vec<i64>used:Vec<bool>busy:Vec<i64>hand:i64stamp:i64hits:i64misses:i64readNs:i64history:Vec<i64>convs:Vec<Vec<f32>>last:Vec<Vec<i64>>ahead:i64
Thought
type Thought = { open: string; close: string; }How a model's reply marks its thinking: open is the text it writes to start a thought ("": the prompt opens it,
as ChatML's <think>), close the text that ends it ("": the model does not think).
open:stringclose:string
Chat
type Chat = { template: Template; names: Vec<string>; values: Vec<Value>; xhigh: bool; }A model's chat template (chat_template.jinja or tokenizer_config.json's chat_template) as Hugging Face renders it,
the special tokens it may name (names, their values: bos_token, eos_token), and whether its top reasoning effort
is called xhigh (Qwen 3.8), where a chat's "high" goes.
template:gizlang_jinja.Templatenames:Vec<string>values:Vec<gizlang_jinja.Value>xhigh:bool
Model
type Model = { arch: Arch; folder: string; tokenizer: Tokenizer; stops: Vec<i64>; chat?: Chat; thought: Thought; call: string; sampling: Sampling; embed: Stored; layers: Vec<Layer>; norm: i64; head: i64; mtp: Option<Mtp>; mixer: Vec<i64>; drafter?: Drafter; linears: Vec<Linear>; small: Vec<Vec<f32>>; gpu: Option<Gpu>; device: Vec<DeviceLinear>; deviceSmall: Vec<Buffer>; signs: Vec<Buffer>; sources: Vec<Vec<string>>; embedTable?: Mapped; layerInput?: { embed: Stored; projection: i64; norm: i64; }; files: FlashFiles; vision?: Vision; weights?: Stack; spots: Vec<Vec<Spot>>; beside: i64; stages: Vec<Model>; kvCosts?: string; npu?: NpuLlm.Model; }A converted model, on the GPU that gizhw/gpu opens (GIZLANG_DEVICE) when there is one (device), else on the
host (linears); the small tensors are on both. The embedding stays in the file, and where the GPU of the first layer
reads it (embedTable, Gpu.mapFile): a step there gathers its tokens' rows on the GPU, the host reads them from the file.
chat its chat template when it has one (Chat),
thought how its reply marks its thinking (Thought),
call the mark where its tool calls start (gizai/llm/model CALL, PYTHON_CALL; "": it writes none); stops are the tokens that end
a generation and sampling the model's own settings. mtp is the MTP layer on a GPU when the model has one, vision
its vision tower when its folder has one (convertVision), stages the models of its later layers on other GPUs
(loadAcross).
arch:_gizlang_lib_ai__llm__shape.Archfolder:stringtokenizer:M26.Tokenizerstops:Vec<i64>chat:_gizlang_sdk_core.Option<M70.Chat>thought:M70.Thoughtcall:stringsampling:M72.Samplingembed:gizlang_safetensors.Storedlayers:Vec<M70.Layer>norm:i64head:i64mtp:_gizlang_sdk_core.Option<M70.Mtp>mixer:Vec<i64>drafter:_gizlang_sdk_core.Option<M70.Drafter>linears:Vec<M70.Linear>small:Vec<Vec<f32>>gpu:_gizlang_sdk_core.Option<M7.Gpu>device:Vec<M70.DeviceLinear>deviceSmall:Vec<M7.Buffer>signs:Vec<M7.Buffer>sources:Vec<Vec<string>>embedTable:_gizlang_sdk_core.Option<M7.Mapped>layerInput:_gizlang_sdk_core.Option<{ embed: gizlang_safetensors.Stored; projection: i64; norm: i64; }>files:M70.FlashFilesvision:_gizlang_sdk_core.Option<_gizlang_lib_ai__llm__vision.Vision>weights:_gizlang_sdk_core.Option<_gizlang_lib_ai__parts__stack.Stack>spots:Vec<Vec<M70.Spot>>beside:i64stages:Vec<M70.Model>kvCosts:_gizlang_sdk_core.Option<string>npu:_gizlang_sdk_core.Option<_gizlang_lib_ai__llm__npu__gemma.Model>
Spot
type Spot = { linear: i64; codes: i64; scales: i64; book: i64; }A linear of a layer's group (Model.weights): its place among the model's device linears, and which of the group's tensors are its codes, its scales and its book.
linear:i64codes:i64scales:i64book:i64
Functions
deltaTerms
function deltaTerms(a: Arch): i64Floats of a draft row's delta rule terms per value head: its normed key, value, decay and beta.
Effects: fail.
descRows
function descRows(rows: i64): i64The descriptor of a step (W_DESC, f32) for rows rows: per row its slot and position, per sequence its slot, first
row, row count and drafts (its last rows, see Engine.generate), per row the rotary cosines and sines, the wanted rows,
then the tiles (first row and rows, see tileKernel), the local rotary table and the rows' tokens (embedKernel).
descSegments
function descSegments(rows: i64): i64Where the per-sequence part of the descriptor starts.
Effects: fail.
descRope
function descRope(rows: i64): i64Where the rows' rotary cosines and sines start.
Effects: fail.
descWanted
function descWanted(a: Arch, rows: i64): i64Where the wanted rows start.
Effects: fail.
descTiles
function descTiles(a: Arch, rows: i64): i64Where the tiles start.
Effects: fail.
descLocal
function descLocal(a: Arch, rows: i64): i64The local layers' rotary table (Arch.local), after the tiles.
Effects: fail.
descTokens
function descTokens(a: Arch, rows: i64): i64Where the rows' tokens start (embedKernel).
Effects: fail.
descSize
function descSize(a: Arch, rows: i64): i64The floats of the whole descriptor.
Effects: fail.
addressAt
function addressAt(b: Buffer, floats: i64): u64The address FLOATS floats into B.
Effects: fail.
partCapacity
function partCapacity(rows: i64): i64The parts the parts buffer holds for an engine of rows rows.
Effects: fail.
partFloats
function partFloats(headDim: i64): i64The floats of a split's partial result of one row and query head (attention's parts, the drafter's too): its weighted values (headDim), their largest score and the sum of their weights.
Effects: fail.
stretches
function stretches(e: Engine, prompt: i64, limit: i64): boolWhether a request of prompt tokens that writes at most limit is long: the model has a long table (Arch.long)
and the request may pass Arch.longFrom. The others turn by the plain table (logits as within the trained context);
one that passes longFrom all the same goes on with it.
Effects: fail.
attentionLayers
function attentionLayers(a: Arch): i64The attention layers of a.
Effects: fail.
recurrentLayers
function recurrentLayers(a: Arch): i64The recurrent (non-attention) layers of a.
Effects: fail.
drafting
function drafting(e: Engine): boolWhether generate drafts with the model's MTP layer (on a model in stages, the last stage's: headOf).
Effects: fail.
looksUp
function looksUp(e: Engine): boolWhether generate drafts by lookup (a model without an MTP layer or block drafter; not on an NPU, which reads a token a step).
Effects: fail.
mtpOn
function mtpOn(e: Engine): boolWhether this engine's own model runs the MTP layer: its buffers and KV pages hold it.
Effects: fail.
poolLayers
function poolLayers(e: Engine): i64Attention layers in the KV pages: the model's, and its MTP layer's when generate drafts with it.
Effects: fail.
pageBytes
function pageBytes(e: Engine): i64The bytes of a KV page: keys and values of PAGE positions in each pool layer (Engine.kv).
poolWides
function poolWides(e: Engine): Vec<bool>Per pool layer (Layer.at): whether its heads are the wide shape (Arch.wideView); the MTP layer's last.
Effects: fail, alloc.
kvGroup
function kvGroup(e: Engine, at: i64): KvGroupThe group of the KV pages that keeps the keys and values pool layer at (Layer.source) reads.
Effects: fail.
hostWidth
function hostWidth(e: Engine, l: i64): i64Floats of a position's keys (or values) in pool layer l (Layer.at) on the host.
Effects: fail.
widestInput
function widestInput(a: Arch): i64Floats of the widest input of a linear and of the widest output.
Effects: fail.
widestOutput
function widestOutput(a: Arch): i64Floats of the widest output of a linear, rounded up to 16.
Effects: fail, alloc.
attentionWidth
function attentionWidth(a: Arch): i64Floats of the widest attention output of a row: heads times the widest head.
Effects: fail.
wantedRows
function wantedRows(e: Engine): i64The most rows of a step whose logits a GPU engine gives: 1 + LEVELS a sequence, and the recalled drafts of one sequence beyond those (Engine.advance recalls for one a step): 1 + depth a sequence would hold 0.5 GB of logits at 8 slots.
Effects: fail.
draftRows
function draftRows(e: Engine): i64The draft rows each recurrent layer of W_DRAFT_CONV and W_DRAFT_DELTA holds (draftBase): a step's drafts are among its wanted rows.
Effects: fail.
headOf
function headOf(e: Engine): EngineThe engine whose model has the head: the last stage's (Engine.stages), else e.
Effects: fail.
noTurn
function noTurn(): TurnA turn of no request: the turn of an empty slot.
Effects: fail, alloc.
noSaved
function noSaved(): SavedNo saved states.
noLora
function noLora(slots: i64): EngineLoraAn engine of slots slots that serves no LoRA.
Effects: fail, alloc.
noFiles
function noFiles(): FlashFilesFlashFiles of no file.
Effects: fail.
noFlash
function noFlash(): FlashThe Flash state of an engine without Flash.
Effects: fail, write, alloc.
noThought
function noThought(): ThoughtThe Thought of a model that does not think.
interleaved
function interleaved(states: Vec<Vec<f32>>, rows: i64, dim: i64): Vec<f32>Per row the rows of each of states (rows of dim floats each), in turn.
Effects: fail, alloc.
embedded
function embedded(e: Engine, tokens: Vec<i64>): Vec<f32> throws stringThe embedding rows of the tokens, read from the model file.
Effects: fail, read, write, console, io, alloc.
ropeTable
function ropeTable(a: Arch, positions: Vec<i64>, grid: Vec<i64> = [], table: i64 = GLOBAL, long: Vec<bool> = []): Vec<f32>The rotary cosines and sines of each row's position in rotary table table (GLOBAL, LOCAL): rotary/2 pairs a row,
on the global table of the long frequencies times Arch.longScale for a row of a long sequence (long, Rows.long),
else times Arch.ropeScale. With a grid (three positions a row: time, height, width) frequency j takes height's where j % 3 is 1 and width's where it is 2, within 3
times their sections (a.sections), time's elsewhere: Qwen3-VL's interleaved multimodal rotary positions.
Effects: fail, alloc.
pleKept
function pleKept(): i64Positions of the PLE convolution's inputs a slot keeps: (taps - 1) times the dilation.
Effects: fail.
blocksOn
function blocksOn(e: Engine): boolWhether this engine drafts with its model's block drafter (Model.drafter): on one GPU.
ringPositions
function ringPositions(d: DraftShape, depth: i64): i64The positions a slot's ring holds for each drafter layer on an engine of depth drafts: the window, and the most
rows a step writes past a sequence's next token (a block's later rows, or recalled drafts), so the rows a step
rejects land where the window no longer reads.
Effects: fail.
Methods
Engine.reset
function Engine.reset(self: mut Engine, slot: i64): voidEmpties a slot for a new sequence.
Effects: fail, read, write, console, io, alloc.
Constants
W_X
const W_X: i64 = 0The rows' hidden states (rows by dim).
W_Y
const W_Y: i64 = 1A product's input, normed (rows by the widest input).
W_P
const W_P: i64 = 2A product's output (rows by the widest output).
W_Q
const W_Q: i64 = 3The rows' queries and attention's output (rows by the attention width).
W_C
const W_C: i64 = 4The convolution's inputs and outputs (rows by convDim).
W_PARTS
const W_PARTS: i64 = 5Attention's partial results of each split (partCapacity).
W_DESC
const W_DESC: i64 = 6The step's descriptor (descSize).
W_MAP
const W_MAP: i64 = 7The slot of each row.
W_TABLE
const W_TABLE: i64 = 8The page table: each slot's KV pages.
W_POOL
const W_POOL: i64 = 9The KV pool: the pages of every slot's keys and values.
W_CONV
const W_CONV: i64 = 10Each slot's last convolution inputs.
W_STATE
const W_STATE: i64 = 11Each slot's delta rule states, BF16 two to a float.
W_SCRATCH
const W_SCRATCH: i64 = 12The partial sums of a split product, or the wanted rows' logits.
W_HID
const W_HID: i64 = 13The last hidden states of a step's rows, normed, for the MTP layer.
W_EMB
const W_EMB: i64 = 14The embeddings of the MTP layer's tokens.
W_SRC
const W_SRC: i64 = 15The step rows whose hidden states the MTP layer reads.
W_LOGITS
const W_LOGITS: i64 = 16The wanted rows' logits.
W_TOP
const W_TOP: i64 = 17The wanted rows' most likely tokens.
W_DRAFT_CONV
const W_DRAFT_CONV: i64 = 18Each slot's convolution inputs of its drafts in the last step (Engine.generate).
W_DRAFT_DELTA
const W_DRAFT_DELTA: i64 = 19Each slot's delta rule terms of its drafts in the last step (deltaTerms).
W_DRAFT_BLOCKS
const W_DRAFT_BLOCKS: i64 = 20The head blocks a draft reads (Engine.draftBlocks).
W_CAND
const W_CAND: i64 = 21The wanted rows' sampling candidates (candidatesKernel).
W_X8
const W_X8: i64 = 22x of an FP8 product (fp8Kernel): its rows' scales (big floats), then its E4M3 values, 4 to a float; or x turned back from its rotation for a K format (plainKernel).
W_LAYER_INPUT
const W_LAYER_INPUT: i64 = 23The rows' per-layer inputs (rows by layers by layerInput), after their projection.
W_LAYER_EMBED
const W_LAYER_EMBED: i64 = 24The rows' per-layer embeddings (rows by layers by layerInput), as the host looked them up.
SPLITS
const SPLITS: i64 = 16Splits of the keys that attention reads at once: a step by tiles takes partSplits; attendKernel (fewer rows) more, up to two waves of its workgroups on the GPU (Gpu.slots; ATTEND_GROUPS where the driver does not say) within the parts buffer (partCapacity), each split of at least SPAN keys. (One row's 4 kv heads in 16 splits kept 64 workgroups on 76 multiprocessors: 60K keys took 0.8 ms a layer, nine times what the memory takes to read them.)
SPAN
const SPAN: i64 = 64The fewest keys of a split.
ATTEND_GROUPS
const ATTEND_GROUPS: i64 = 512The workgroups of attention's two waves where the driver does not say (Gpu.slots).
LOG2E
const LOG2E = Math.LOG2Elog2 e: softmax weights are 2^(x log2 e) on the GPU's exp2 (its exp polynomial cost some 25 instructions a weight).
GEMV_ROWS
const GEMV_ROWS: i64 = 8The most rows a GPU linear reads its weights for at once on CUDA cores; from GEMM_ROWS rows a linear runs on matrix cores where the GPU has them.
GEMM_ROWS
const GEMM_ROWS: i64 = 32The fewest rows a linear runs on matrix cores for.
PACE_CLASSES
const PACE_CLASSES: i64 = 4The classes of context length Drafting.steps keeps apart (paceClass).
DRAFTS
const DRAFTS: i64 = 32The drafts an engine reads per sequence and step by default: recalled drafts (LlmGenerate.recalled) take all of them; those of the MTP layer or lookup at most LEVELS, as the n-th is kept less often than the one before and each costs a row (and with an MTP layer a step of it and the head).
LEVELS
const LEVELS: i64 = 8The most drafts an MTP layer or lookup makes a step (LlmGenerate.draftLevels).
DRAFT_SHARE
const DRAFT_SHARE: i64 = 8A draft reads one DRAFT_SHARE-th of the head's blocks of 16 rows (as FR-Spec, arXiv 2502.14856, which takes the frequent tokens of a corpus): the first half of them always (a BPE vocabulary's first ids are its frequent tokens), the other half the last blocks of the tokens the engine read, so the language of a prompt comes in with it.
MTP_WINDOW
const MTP_WINDOW: i64 = 32768The last keys the MTP layer attends to: its drafts lean on the near text, and the model checks every one, so the output is the same; at 252K a draft pass reads an eighth of the keys.
RECORDINGS
const RECORDINGS: i64 = 64The most step shapes an engine keeps recorded.
RECORD_ROOM
const RECORD_ROOM: i64 = 50_331_648The device memory free before a new recording at least (one of q27-4's steps took 6 MiB).
PLE_TAPS
const PLE_TAPS: i64 = 4The taps of the PLE convolution.
PLE_DILATION
const PLE_DILATION: i64 = 3The dilation of the PLE convolution.
SPLIT_ROOM
const SPLIT_ROOM: i64 = 1_073_741_824The device memory a model's weights leave free on its GPU at least, for an engine's work buffers and KV pages: a model that leaves less takes the GPU's other GPUs too (loadAcross), or keeps layers in host memory (Model.weights).