gizai:llm/engine

the Engine, which runs steps of many sequences at once: its slots, KV pages, scheduler state and devices.

import * as Engine from "gizai:llm/engine";

Types

Engine

type Engine = { model: Model; slots: i64; context: i64; rows: i64; lengths: Vec<i64>; kv: KvFormat; host: HostState; device: DeviceState; draft: Drafting; queue: Queue; prefix: PrefixCache; pieces: Vec<bytes>; flash: Flash; stages: Vec<Engine>; exact: bool; }

The engine of a model: slots sequences of at most context positions, steps of at most rows rows.

• model: M51.Model
• slots: i64
• context: i64
• rows: i64
• lengths: Vec<i64>
• kv: _gizlang_lib_ai__llm__kvformat.KvFormat
• host: M50.HostState
• device: M50.DeviceState
• draft: M50.Drafting
• queue: M50.Queue
• prefix: M50.PrefixCache
• pieces: Vec<bytes> — The bytes of each token (Tokenizer.tokenBytes), read at the first request with a grammar.
• flash: M51.Flash
• stages: Vec<M50.Engine>
• exact: bool — Linears give x to the GEMMs as the F32 the model computes (rounded to F16 there): no gemm16Kernel, whose 16-bit weights take x in their own type (BF16 for BF16 weights, as other servers' activations). Off: the faster.

Rows

type Rows = { tokens: Vec<i64>; slots: Vec<i64>; positions: Vec<i64>; ropes: Vec<i64>; grid: Vec<i64>; own: i64; wanted: Vec<i64>; segments: Vec<i64>; drafts: Vec<i64>; layerInputs?: Vec<f32>; }

A step: per row its token, slot, position and rotary position (or three: grid, Visual.grid), the rows whose logits are wanted, per sequence its slot, first row and row count, and per sequence its drafts (its last rows, see Engine.generate). A row sees the keys before its position, and its own when own is 1 (0: a padding row, see Engine.padding).

• tokens: Vec<i64>
• slots: Vec<i64>
• positions: Vec<i64>
• ropes: Vec<i64>
• grid: Vec<i64>
• own: i64
• wanted: Vec<i64>
• segments: Vec<i64>
• drafts: Vec<i64>
• layerInputs: _gizlang_sdk_core.Option<Vec<f32>>

Visual

type Visual = { rows: Vec<i64>; embeds: Vec<f32>; deep: Vec<Vec<f32>>; grid: Vec<i64>; }

Rows of a step that hold no token's embedding (Qwen3-VL's pictures): rows of the step, their embeddings (dim floats a row, in turn), deep the features added to them after each of the first layers (Qwen3-VL's deepstack, rows as embeds), and grid every row's three rotary positions (time, height, width; a text row's three are its position: see ropeTable).

• rows: Vec<i64>
• embeds: Vec<f32>
• deep: Vec<Vec<f32>>
• grid: Vec<i64>

Functions

partCapacity

function partCapacity(rows: i64): i64

The parts the parts buffer holds for an engine of rows rows.

Effects: fail.

partSplits

function partSplits(e: Engine, n: i64): i64

The most splits of a row, a power of two of at most SPLITS, whose parts of n rows the parts buffer holds.

Effects: fail.

attendSplits

function attendSplits(e: Engine, g: Gpu, n: i64): i64

The splits of a row that attendKernel takes for a step of n rows on g.

Effects: fail, read, write, console, io.

attentionLayers

function attentionLayers(a: Arch): i64

The attention layers of a.

Effects: fail.

recurrentLayers

function recurrentLayers(a: Arch): i64

The recurrent (non-attention) layers of a.

Effects: fail.

drafting

function drafting(e: Engine): bool

Whether generate drafts with the model's MTP layer (on a model in stages, the last stage's: headOf).

Effects: fail.

looksUp

function looksUp(e: Engine): bool

Whether generate drafts by lookup (a model without an MTP layer; not on an NPU, which reads a token a step).

Effects: fail.

mtpOn

function mtpOn(e: Engine): bool

Whether this engine's own model runs the MTP layer: its buffers and KV pages hold it.

Effects: fail.

poolLayers

function poolLayers(e: Engine): i64

Attention layers in the KV pages: the model's, and its MTP layer's when generate drafts with it.

Effects: fail.

pageBytes

function pageBytes(e: Engine): i64

The bytes of a KV page: keys and values of PAGE positions in each pool layer.

Effects: fail.

layerArch

function layerArch(e: Engine, wide: bool): Arch

The Arch of a layer's attention: its head shape's with its region of the KV pages (kvViews).

Effects: fail.

hostWidth

function hostWidth(e: Engine, l: i64): i64

Floats of a position's keys (or values) in pool layer l (Layer.at) on the host.

Effects: fail.

shapeLayers

function shapeLayers(e: Engine, wide: bool): i64

The pool layers of the wide head shape, or of the other one.

Effects: fail.

widestInput

function widestInput(a: Arch): i64

Floats of the widest input of a linear and of the widest output.

Effects: fail.

widestOutput

function widestOutput(a: Arch): i64

Floats of the widest output of a linear, rounded up to 16.

Effects: fail.

attentionWidth

function attentionWidth(a: Arch): i64

Floats of the widest attention output of a row: heads times the widest head.

Effects: fail.

wantedRows

function wantedRows(e: Engine): i64

The most rows of a step whose logits a GPU engine gives.

Effects: fail.

forward

function forward(e: mut Engine, sys: System, tokens: Vec<i64>, slots: Vec<i64>, wanted: Vec<i64>, drafts: Vec<i64> = [], taps: Vec<i64> = [], padding: i64 = -1, visual: Option<Visual> = undefined): Forward throws string

drafts: per slot its drafts in the step, its last rows (none when empty; see Engine.step). taps: the layers after which the step keeps the residual stream (Engine.hidden). padding: the rotary position of the step's one row, which then sees the slot's keys but not its own and leaves the slot as it was (Engine.padding; -1: none).

Effects: fail, read, write, console, io.

interleaved

function interleaved(states: Vec<Vec<f32>>, rows: i64, dim: i64): Vec<f32>

Per row the rows of each of states (rows of dim floats each), in turn.

Effects: fail.

embedded

function embedded(e: Engine, sys: System, tokens: Vec<i64>): Vec<f32> throws string

The embedding rows of the tokens, read from the model file.

Effects: fail, io.

ropeTable

function ropeTable(a: Arch, positions: Vec<i64>, grid: Vec<i64> = [], table: i64 = GLOBAL): Vec<f32>

The rotary cosines and sines of each row's position in rotary table table (GLOBAL, LOCAL): rotary/2 pairs a row, times Arch.ropeScale on the global table, of the long frequencies from Arch.longFrom on. With a grid (three positions a row: time, height, width) frequency j takes height's where j % 3 is 1 and width's where it is 2, within 3 times their sections (a.sections), time's elsewhere: Qwen3-VL's interleaved multimodal rotary positions.

Effects: fail.

Methods

Engine.reset

function Engine.reset(self: mut Engine, slot: i64): void

Empties a slot for a new sequence.

Effects: fail, read, write, console, io.

Engine.length

function Engine.length(self: Engine, slot: i64): i64

Positions held by a slot.

Effects: fail.

Engine.step

function Engine.step(self: mut Engine, sys: System, tokens: Vec<i64>, slots: Vec<i64>, wanted: Vec<i64>, drafts: Vec<i64> = [], visual: Option<Visual> = undefined): Vec<f32> throws string

Reads tokens into their slots (the rows of a slot together and in order, after what the slot holds) and returns the logits of the rows wanted (vocab floats each, in that order). On an engine that drafts (Engine.depth), a slot's last drafts[slot] rows (none when drafts is empty) are drafts of the tokens after its row before them: they wait for Engine.accept, which the slot's next step needs first. With visual, some rows are a picture's (Visual, as Engine.hidden), and every row takes its three rotary positions from it.

Effects: fail, read, write, console, io.

Engine.accept

function Engine.accept(self: mut Engine, slot: i64, kept: i64): bool throws string

Keeps the first kept of a slot's drafts of its last step and drops the others.

Effects: fail, read, write, console, io.

Engine.hidden

function Engine.hidden(self: mut Engine, sys: System, tokens: Vec<i64>, slots: Vec<i64>, layers: Vec<i64> = [], visual: Option<Visual> = undefined): Vec<f32> throws string

Reads tokens into their slots as Engine.step does and returns the residual stream of each row after each of layers (counted from 1; none: the model's last), dim floats a row and layer: Hugging Face's hidden_states[l] for each l, before the final norm (Hugging Face norms the whole model's last hidden state). With visual, some rows are a picture's (Visual), not their tokens' embeddings.

Effects: fail, read, write, console, io.

Engine.padding

function Engine.padding(self: mut Engine, sys: System, slot: i64, token: i64, position: i64, layers: Vec<i64> = []): Vec<f32> throws string

The hidden states (as Engine.hidden) of one row of token at rotary position position that sees the keys of the slot's tokens but not its own, and leaves the slot as it was: a padding row of a Hugging Face batch, whose mask hides the padding keys from every row. Every padding row of a sequence is this one row (FLUX.3 reads Qwen3-VL's, at position 1). Models of attention layers only.

Effects: fail, read, write, console, io.

Engine.expertCache

function Engine.expertCache(self: Engine): ExpertCache

The expert cache counts of the engine.

Engine.generate

function Engine.generate(self: mut Engine, sys: System, requests: Vec<Request>): Vec<Vec<i64>> throws string

Generates for every request and returns each one's new tokens (without the stop token): Engine.add of each, then Engine.advance until none is in turn.

Effects: fail, read, write, console, io.

Engine.warm

function Engine.warm(self: mut Engine, sys: System): bool throws string

Runs once each kind of step a server meets first: a long prompt read in parts, a prompt of each size its linears tune by (delta.giz linear: powers of 2 below GEMM_ROWS, then 128 rows each up to the step's rows), then 1 to slots sequences that read a prompt and write tokens. The first run of a step shape compiles, tunes and records its kernels (seconds for a large model), so a server warms before it takes requests.

Effects: fail, read, write, console, io.

Engine.add

function Engine.add(self: mut Engine, request0: Request, mark: i64 = 0, head: i64 = 0): i64 throws string

Puts a request in turn and returns its id, which Engine.advance's results name. A free slot that holds the start of its prompt (an earlier request's prompt and answer) takes it and reads only the rest; so does a slot that takes back such a start from the host, where a slot keeps what it held when another request takes it (Prefix). After mark tokens of the prompt (0: none) its slot also keeps the recurrent states, so a later prompt that starts with those tokens but not with all the slot read reads only the rest too; for a chat that is where its messages end (Model.chatTokens). The prompt's first head tokens (0: none; for a chat its system turn with the tools), cut to whole KV pages and when at least SHARED_MIN, become a start that later prompts share (Shared): a slot whose prompt starts with it reads only the rest, and on a GPU all such slots read one copy of its keys and values.

Effects: fail.

Engine.cancel

function Engine.cancel(self: mut Engine, id: i64): bool

Takes a request out of turn; false when it is not in turn (it ended). Its slot keeps what it read.

Effects: fail.

Engine.busy

function Engine.busy(self: Engine): bool

Whether requests are in turn.

Effects: fail.

Engine.advance

function Engine.advance(self: mut Engine, sys: System): Vec<Progress> throws string

One step of the requests in turn, and what each gained. First free slots take waiting requests (Engine.add). The step reads the next token of every generating sequence and, in the rows left, the next piece of the prompts not read yet. When the KV pool has no room for the step, a free slot gives its pages, else the request that came last waits again from where it is (its prompt and answer so far).

A drafting engine (Engine.depth) also reads each generating sequence's drafts of the tokens after that: each row judges the draft after it (kept when it is the token picked; when sampling, kept with its probability and otherwise a token other than it is picked, which gives the same distribution as without drafts) up to the first rejected, and when all are kept the last row gives one more token, so a step reading weights once can give 1 + depth tokens. The step reads them as drafts (Engine.step) and keeps those judged right (Engine.accept). Then the MTP layer reads every row kept, with the token after it, and where drafts gave tokens faster for as many generating sequences (EXPLORE), drafts after each sequence's last; it reads each draft again, with its own hidden state for it, for the next.

Effects: fail, read, write, console, io.

Engine.complete

function Engine.complete(self: mut Engine, sys: System, prompt: string, limit: i64, s: Sampling): string throws string

The text the model writes after prompt (special tokens in it read as such), at most limit tokens.

Effects: fail, read, write, console, io.

Constants

SPLITS

const SPLITS = 16

Splits of the keys that attention reads at once: a step by tiles takes partSplits; attendKernel (fewer rows) more, up to two waves of its workgroups on the GPU (Gpu.slots; ATTEND_GROUPS where the driver does not say) within the parts buffer (partCapacity), each split of at least SPAN keys. (One row's 4 kv heads in 16 splits kept 64 workgroups on 76 multiprocessors: 60K keys took 0.8 ms a layer, nine times what the memory takes to read them.)

SPAN

const SPAN = 64

The fewest keys of a split.

ATTEND_GROUPS

const ATTEND_GROUPS = 512

The workgroups of attention's two waves where the driver does not say (Gpu.slots).

PART_ROWS

const PART_ROWS = 256

The parts buffer (W_PARTS) holds SPLITS parts of PART_ROWS rows (or of the engine's rows when fewer), and at least one part of each row: a step of more rows takes fewer splits a row (partSplits).

GEMV_ROWS

const GEMV_ROWS = 8

The most rows a GPU linear reads its weights for at once on CUDA cores; from GEMM_ROWS rows a linear runs on matrix cores where the GPU has them.

GEMM_ROWS

const GEMM_ROWS = 32

The fewest rows a linear runs on matrix cores for.

EXPLORE

const EXPLORE = 256

An engine drafts with an MTP layer, or by lookup without one, where that gives tokens faster (Engine.paces): drafting pays where the GPU waits on weights (few sequences), and a step with drafts reads more rows and reads and writes a slot's recurrent states once more (replayDrafts). Every EXPLORE steps each count of sequences runs the other way twice, to measure it again.

DRAFTS

const DRAFTS = 8

The tokens an engine drafts per sequence and step by default: the n-th draft is kept less often than the one before, and each costs a row (and with an MTP layer a step of it and the head).

DRAFT_SHARE

const DRAFT_SHARE = 8

A draft reads one DRAFT_SHARE-th of the head's blocks of 16 rows (as FR-Spec, arXiv 2502.14856, which takes the frequent tokens of a corpus): the first half of them always (a BPE vocabulary's first ids are its frequent tokens), the other half the last blocks of the tokens the engine read, so the language of a prompt comes in with it.

RECORDINGS

const RECORDINGS = 64

The most step shapes an engine keeps recorded.

RECORD_ROOM

const RECORD_ROOM = 50_331_648

The device memory free before a new recording at least (one of q27-4's steps took 6 MiB).