# gizai:llm/engine the Engine, which runs steps of many sequences at once: its slots, KV pages, scheduler state and devices. ```gizlang import * as Engine from "gizai:llm/engine"; ``` ## Types ### `Engine` ```gizlang type Engine = { model: Model; slots: i64; context: i64; rows: i64; lengths: Vec; kv: KvFormat; host: HostState; device: DeviceState; draft: Drafting; queue: Queue; prefix: PrefixCache; pieces: Vec; flash: Flash; stages: Vec; exact: bool; } ``` The engine of a model: `slots` sequences of at most `context` positions, steps of at most `rows` rows. - `model`: `M51.Model` - `slots`: `i64` - `context`: `i64` - `rows`: `i64` - `lengths`: `Vec` - `kv`: `_gizlang_lib_ai__llm__kvformat.KvFormat` - `host`: `M50.HostState` - `device`: `M50.DeviceState` - `draft`: `M50.Drafting` - `queue`: `M50.Queue` - `prefix`: `M50.PrefixCache` - `pieces`: `Vec` — The bytes of each token (Tokenizer.tokenBytes), read at the first request with a grammar. - `flash`: `M51.Flash` - `stages`: `Vec` - `exact`: `bool` — Linears give x to the GEMMs as the F32 the model computes (rounded to F16 there): no gemm16Kernel, whose 16-bit weights take x in their own type (BF16 for BF16 weights, as other servers' activations). Off: the faster. ### `Rows` ```gizlang type Rows = { tokens: Vec; slots: Vec; positions: Vec; ropes: Vec; grid: Vec; own: i64; wanted: Vec; segments: Vec; drafts: Vec; layerInputs?: Vec; } ``` A step: per row its token, slot, position and rotary position (or three: grid, Visual.grid), the rows whose logits are wanted, per sequence its slot, first row and row count, and per sequence its drafts (its last rows, see Engine.generate). A row sees the keys before its position, and its own when `own` is 1 (0: a padding row, see Engine.padding). - `tokens`: `Vec` - `slots`: `Vec` - `positions`: `Vec` - `ropes`: `Vec` - `grid`: `Vec` - `own`: `i64` - `wanted`: `Vec` - `segments`: `Vec` - `drafts`: `Vec` - `layerInputs`: `_gizlang_sdk_core.Option>` ### `Visual` ```gizlang type Visual = { rows: Vec; embeds: Vec; deep: Vec>; grid: Vec; } ``` Rows of a step that hold no token's embedding (Qwen3-VL's pictures): `rows` of the step, their embeddings (dim floats a row, in turn), `deep` the features added to them after each of the first layers (Qwen3-VL's deepstack, rows as embeds), and `grid` every row's three rotary positions (time, height, width; a text row's three are its position: see ropeTable). - `rows`: `Vec` - `embeds`: `Vec` - `deep`: `Vec>` - `grid`: `Vec` ## Functions ### `partCapacity` ```gizlang function partCapacity(rows: i64): i64 ``` The parts the parts buffer holds for an engine of `rows` rows. Effects: fail. ### `partSplits` ```gizlang function partSplits(e: Engine, n: i64): i64 ``` The most splits of a row, a power of two of at most SPLITS, whose parts of n rows the parts buffer holds. Effects: fail. ### `attendSplits` ```gizlang function attendSplits(e: Engine, g: Gpu, n: i64): i64 ``` The splits of a row that attendKernel takes for a step of `n` rows on `g`. Effects: fail, read, write, console, io. ### `attentionLayers` ```gizlang function attentionLayers(a: Arch): i64 ``` The attention layers of `a`. Effects: fail. ### `recurrentLayers` ```gizlang function recurrentLayers(a: Arch): i64 ``` The recurrent (non-attention) layers of `a`. Effects: fail. ### `drafting` ```gizlang function drafting(e: Engine): bool ``` Whether generate drafts with the model's MTP layer (on a model in stages, the last stage's: headOf). Effects: fail. ### `looksUp` ```gizlang function looksUp(e: Engine): bool ``` Whether generate drafts by lookup (a model without an MTP layer; not on an NPU, which reads a token a step). Effects: fail. ### `mtpOn` ```gizlang function mtpOn(e: Engine): bool ``` Whether this engine's own model runs the MTP layer: its buffers and KV pages hold it. Effects: fail. ### `poolLayers` ```gizlang function poolLayers(e: Engine): i64 ``` Attention layers in the KV pages: the model's, and its MTP layer's when generate drafts with it. Effects: fail. ### `pageBytes` ```gizlang function pageBytes(e: Engine): i64 ``` The bytes of a KV page: keys and values of PAGE positions in each pool layer. Effects: fail. ### `layerArch` ```gizlang function layerArch(e: Engine, wide: bool): Arch ``` The Arch of a layer's attention: its head shape's with its region of the KV pages (kvViews). Effects: fail. ### `hostWidth` ```gizlang function hostWidth(e: Engine, l: i64): i64 ``` Floats of a position's keys (or values) in pool layer l (Layer.at) on the host. Effects: fail. ### `shapeLayers` ```gizlang function shapeLayers(e: Engine, wide: bool): i64 ``` The pool layers of the wide head shape, or of the other one. Effects: fail. ### `widestInput` ```gizlang function widestInput(a: Arch): i64 ``` Floats of the widest input of a linear and of the widest output. Effects: fail. ### `widestOutput` ```gizlang function widestOutput(a: Arch): i64 ``` Floats of the widest output of a linear, rounded up to 16. Effects: fail. ### `attentionWidth` ```gizlang function attentionWidth(a: Arch): i64 ``` Floats of the widest attention output of a row: heads times the widest head. Effects: fail. ### `wantedRows` ```gizlang function wantedRows(e: Engine): i64 ``` The most rows of a step whose logits a GPU engine gives. Effects: fail. ### `forward` ```gizlang function forward(e: mut Engine, sys: System, tokens: Vec, slots: Vec, wanted: Vec, drafts: Vec = [], taps: Vec = [], padding: i64 = -1, visual: Option = undefined): Forward throws string ``` drafts: per slot its drafts in the step, its last rows (none when empty; see Engine.step). taps: the layers after which the step keeps the residual stream (Engine.hidden). padding: the rotary position of the step's one row, which then sees the slot's keys but not its own and leaves the slot as it was (Engine.padding; -1: none). Effects: fail, read, write, console, io. ### `interleaved` ```gizlang function interleaved(states: Vec>, rows: i64, dim: i64): Vec ``` Per row the rows of each of `states` (rows of dim floats each), in turn. Effects: fail. ### `embedded` ```gizlang function embedded(e: Engine, sys: System, tokens: Vec): Vec throws string ``` The embedding rows of the tokens, read from the model file. Effects: fail, io. ### `ropeTable` ```gizlang function ropeTable(a: Arch, positions: Vec, grid: Vec = [], table: i64 = GLOBAL): Vec ``` The rotary cosines and sines of each row's position in rotary table `table` (GLOBAL, LOCAL): rotary/2 pairs a row, times Arch.ropeScale on the global table, of the long frequencies from Arch.longFrom on. With a `grid` (three positions a row: time, height, width) frequency j takes height's where j % 3 is 1 and width's where it is 2, within 3 times their sections (a.sections), time's elsewhere: Qwen3-VL's interleaved multimodal rotary positions. Effects: fail. ## Methods ### `Engine.reset` ```gizlang function Engine.reset(self: mut Engine, slot: i64): void ``` Empties a slot for a new sequence. Effects: fail, read, write, console, io. ### `Engine.length` ```gizlang function Engine.length(self: Engine, slot: i64): i64 ``` Positions held by a slot. Effects: fail. ### `Engine.step` ```gizlang function Engine.step(self: mut Engine, sys: System, tokens: Vec, slots: Vec, wanted: Vec, drafts: Vec = [], visual: Option = undefined): Vec throws string ``` Reads `tokens` into their slots (the rows of a slot together and in order, after what the slot holds) and returns the logits of the rows `wanted` (vocab floats each, in that order). On an engine that drafts (Engine.depth), a slot's last drafts[slot] rows (none when drafts is empty) are drafts of the tokens after its row before them: they wait for Engine.accept, which the slot's next step needs first. With `visual`, some rows are a picture's (Visual, as Engine.hidden), and every row takes its three rotary positions from it. Effects: fail, read, write, console, io. ### `Engine.accept` ```gizlang function Engine.accept(self: mut Engine, slot: i64, kept: i64): bool throws string ``` Keeps the first `kept` of a slot's drafts of its last step and drops the others. Effects: fail, read, write, console, io. ### `Engine.hidden` ```gizlang function Engine.hidden(self: mut Engine, sys: System, tokens: Vec, slots: Vec, layers: Vec = [], visual: Option = undefined): Vec throws string ``` Reads `tokens` into their slots as Engine.step does and returns the residual stream of each row after each of `layers` (counted from 1; none: the model's last), dim floats a row and layer: Hugging Face's hidden_states[l] for each l, before the final norm (Hugging Face norms the whole model's last hidden state). With `visual`, some rows are a picture's (Visual), not their tokens' embeddings. Effects: fail, read, write, console, io. ### `Engine.padding` ```gizlang function Engine.padding(self: mut Engine, sys: System, slot: i64, token: i64, position: i64, layers: Vec = []): Vec throws string ``` The hidden states (as Engine.hidden) of one row of `token` at rotary position `position` that sees the keys of the slot's tokens but not its own, and leaves the slot as it was: a padding row of a Hugging Face batch, whose mask hides the padding keys from every row. Every padding row of a sequence is this one row (FLUX.3 reads Qwen3-VL's, at position 1). Models of attention layers only. Effects: fail, read, write, console, io. ### `Engine.expertCache` ```gizlang function Engine.expertCache(self: Engine): ExpertCache ``` The expert cache counts of the engine. ### `Engine.generate` ```gizlang function Engine.generate(self: mut Engine, sys: System, requests: Vec): Vec> throws string ``` Generates for every request and returns each one's new tokens (without the stop token): Engine.add of each, then Engine.advance until none is in turn. Effects: fail, read, write, console, io. ### `Engine.warm` ```gizlang function Engine.warm(self: mut Engine, sys: System): bool throws string ``` Runs once each kind of step a server meets first: a long prompt read in parts, a prompt of each size its linears tune by (delta.giz linear: powers of 2 below GEMM_ROWS, then 128 rows each up to the step's rows), then 1 to `slots` sequences that read a prompt and write tokens. The first run of a step shape compiles, tunes and records its kernels (seconds for a large model), so a server warms before it takes requests. Effects: fail, read, write, console, io. ### `Engine.add` ```gizlang function Engine.add(self: mut Engine, request0: Request, mark: i64 = 0, head: i64 = 0): i64 throws string ``` Puts a request in turn and returns its id, which Engine.advance's results name. A free slot that holds the start of its prompt (an earlier request's prompt and answer) takes it and reads only the rest; so does a slot that takes back such a start from the host, where a slot keeps what it held when another request takes it (Prefix). After `mark` tokens of the prompt (0: none) its slot also keeps the recurrent states, so a later prompt that starts with those tokens but not with all the slot read reads only the rest too; for a chat that is where its messages end (Model.chatTokens). The prompt's first `head` tokens (0: none; for a chat its system turn with the tools), cut to whole KV pages and when at least SHARED_MIN, become a start that later prompts share (Shared): a slot whose prompt starts with it reads only the rest, and on a GPU all such slots read one copy of its keys and values. Effects: fail. ### `Engine.cancel` ```gizlang function Engine.cancel(self: mut Engine, id: i64): bool ``` Takes a request out of turn; false when it is not in turn (it ended). Its slot keeps what it read. Effects: fail. ### `Engine.busy` ```gizlang function Engine.busy(self: Engine): bool ``` Whether requests are in turn. Effects: fail. ### `Engine.advance` ```gizlang function Engine.advance(self: mut Engine, sys: System): Vec throws string ``` One step of the requests in turn, and what each gained. First free slots take waiting requests (Engine.add). The step reads the next token of every generating sequence and, in the rows left, the next piece of the prompts not read yet. When the KV pool has no room for the step, a free slot gives its pages, else the request that came last waits again from where it is (its prompt and answer so far). A drafting engine (Engine.depth) also reads each generating sequence's drafts of the tokens after that: each row judges the draft after it (kept when it is the token picked; when sampling, kept with its probability and otherwise a token other than it is picked, which gives the same distribution as without drafts) up to the first rejected, and when all are kept the last row gives one more token, so a step reading weights once can give 1 + depth tokens. The step reads them as drafts (Engine.step) and keeps those judged right (Engine.accept). Then the MTP layer reads every row kept, with the token after it, and where drafts gave tokens faster for as many generating sequences (EXPLORE), drafts after each sequence's last; it reads each draft again, with its own hidden state for it, for the next. Effects: fail, read, write, console, io. ### `Engine.complete` ```gizlang function Engine.complete(self: mut Engine, sys: System, prompt: string, limit: i64, s: Sampling): string throws string ``` The text the model writes after `prompt` (special tokens in it read as such), at most `limit` tokens. Effects: fail, read, write, console, io. ## Constants ### `SPLITS` ```gizlang const SPLITS = 16 ``` Splits of the keys that attention reads at once: a step by tiles takes partSplits; attendKernel (fewer rows) more, up to two waves of its workgroups on the GPU (Gpu.slots; ATTEND_GROUPS where the driver does not say) within the parts buffer (partCapacity), each split of at least SPAN keys. (One row's 4 kv heads in 16 splits kept 64 workgroups on 76 multiprocessors: 60K keys took 0.8 ms a layer, nine times what the memory takes to read them.) ### `SPAN` ```gizlang const SPAN = 64 ``` The fewest keys of a split. ### `ATTEND_GROUPS` ```gizlang const ATTEND_GROUPS = 512 ``` The workgroups of attention's two waves where the driver does not say (Gpu.slots). ### `PART_ROWS` ```gizlang const PART_ROWS = 256 ``` The parts buffer (W_PARTS) holds SPLITS parts of PART_ROWS rows (or of the engine's rows when fewer), and at least one part of each row: a step of more rows takes fewer splits a row (partSplits). ### `GEMV_ROWS` ```gizlang const GEMV_ROWS = 8 ``` The most rows a GPU linear reads its weights for at once on CUDA cores; from GEMM_ROWS rows a linear runs on matrix cores where the GPU has them. ### `GEMM_ROWS` ```gizlang const GEMM_ROWS = 32 ``` The fewest rows a linear runs on matrix cores for. ### `EXPLORE` ```gizlang const EXPLORE = 256 ``` An engine drafts with an MTP layer, or by lookup without one, where that gives tokens faster (Engine.paces): drafting pays where the GPU waits on weights (few sequences), and a step with drafts reads more rows and reads and writes a slot's recurrent states once more (replayDrafts). Every EXPLORE steps each count of sequences runs the other way twice, to measure it again. ### `DRAFTS` ```gizlang const DRAFTS = 8 ``` The tokens an engine drafts per sequence and step by default: the n-th draft is kept less often than the one before, and each costs a row (and with an MTP layer a step of it and the head). ### `DRAFT_SHARE` ```gizlang const DRAFT_SHARE = 8 ``` A draft reads one DRAFT_SHARE-th of the head's blocks of 16 rows (as FR-Spec, arXiv 2502.14856, which takes the frequent tokens of a corpus): the first half of them always (a BPE vocabulary's first ids are its frequent tokens), the other half the last blocks of the tokens the engine read, so the language of a prompt comes in with it. ### `RECORDINGS` ```gizlang const RECORDINGS = 64 ``` The most step shapes an engine keeps recorded. ### `RECORD_ROOM` ```gizlang const RECORD_ROOM = 50_331_648 ``` The device memory free before a new recording at least (one of q27-4's steps took 6 MiB).