gizai/llm/kvformat
One owner for KV page packing and quantization layout, independent of execution and scheduling.
import * as Kvformat from "gizai/llm/kvformat";Types
KvFormat
type KvFormat = { keys: i64; values: i64; }How an engine keeps keys and values: the code of a key and of a value. 16: F16 values; 8, 4 or 2: codes of the head rotated by blocks of kvBlock, at 2^bits even steps (kvStore); NV4 (NVFP4): E2M1 codes of the rotated head, which the FP4 matrix cores take with no decoding. NV4 keys have an E4M3 scale a block of 16 dims and an F32 scale a head; NV4 values (with NV4 keys only) an F32 scale a head, their codes kept by dims (KvPart.columns).
keys:i64values:i64
KvPart
type KvPart = { bits: i64; word: i64; each: i64; at: i64; page: i64; steps: i64; floats: i64; nv: bool; scales: i64; columns: bool; }Where a KV page keeps one kind (keys or values) of its heads, each pool layer, kv head and position (kvHead): F16
values, or 32 lanes of codes, lane l the codes of values l hd/32 on; each words of type word a lane, from word
at of the page (page words of that type). With codes each head's step and least value (F32) from float steps.
With nv (NV4) a head's F32 scale is its first float there. NV4 keys: the E4M3 block scales hd/64 words (4 a word)
a head from float scales, and a head's 16-byte chunk c at chunk c ^ (position % (hd / 32)) of its row (kvLane), so
the rows a matrix load reads together meet distinct banks. NV4 values (columns): a page's PAGE heads of a pool layer
and kv head by dims, dim d in words 2 d and 2 d + 1 (I32), position p's code in nibble p % 8 of word p / 8: the B
tiles of the FP4 products with the weights (flashNvKernel).
bits:i64word:i64each:i64at:i64page:i64steps:i64floats:i64nv:boolscales:i64columns:bool
KvLayout
type KvLayout = { keys: KvPart; values: KvPart; bytes: i64; }A page's keys and values of the layers pool layers of a's head shape (KvPart): the keys' heads, the values'
heads, their steps, then the block scales of NV4 keys; bytes their size. They start at byte Arch.kvFrom of a shared
page (kvViews).
keys:M65.KvPartvalues:M65.KvPartbytes:i64
KvGroup
type KvGroup = { arch: Arch; format: KvFormat; layers: i64; }The pool layers of one head shape and one format: their Arch with their region of a KV page (Arch.kvFrom, kvPage) and how many they are.
arch:_gizlang_lib_ai__llm__shape.Archformat:M65.KvFormatlayers:i64
KvPlace
type KvPlace = { group: i64; index: i64; }A pool layer's group and its index among the group's layers.
group:i64index:i64
KvPlan
type KvPlan = { formats: Vec<KvFormat>; groups: Vec<KvGroup>; places: Vec<KvPlace>; bytes: i64; }How a KV page keeps each pool layer: its format (fitted, kvFitted), its group and its index there; the groups'
regions one after the other, bytes the page. One group (one shape, one format) takes the whole page.
formats:Vec<M65.KvFormat>groups:Vec<M65.KvGroup>places:Vec<M65.KvPlace>bytes:i64
Functions
kvFormats
function kvFormats(): Vec<KvFormat>The formats an engine chooses from for each pool layer (kvChoice), most bits first. A key's error moves all scores of its position, a value's only its share of the sum: values lose bits first.
Effects: fail, alloc.
kvPrior
function kvPrior(): Vec<f64>A pool layer's cost in each of kvFormats where its model has no measured costs (kvCosts): the KL of Qwen3.8-27B's next tokens with all its 16 pool layers in the format, a 16th each; nv4 is a guess between nv4/4 and 4 until it is measured.
Effects: fail, alloc.
kvCodeBits
function kvCodeBits(code: i64): i64The bits a code takes for each value (NV4: 4, its scales apart).
kvAllowed
function kvAllowed(codes: string): Vec<bool> throws stringWhich kinds of codes a format list keeps (an engine's kvCodes): "all", "nv4" (no 4-bit keys at even steps: the FP4 matrix cores take every 4-bit key) or "uniform" (no NV4).
Effects: fail, alloc.
kvChoice
function kvChoice(a: Arch, wides: Vec<bool>, costs: Vec<Vec<f64>>, room: i64, allowed: Vec<bool>): Vec<KvFormat>The formats of the pool layers (head shapes wides, as kvPlan) of least summed cost (costs: a row a pool layer,
a cost for each of kvFormats) in which a page takes at most room bytes, each from the kvFormats that allowed
marks; [] where none fits. An exact knapsack: the costs of each layer's change add up (as AutoQuant KV takes them).
Effects: fail, alloc.
kvFormat
function kvFormat(name: string): KvFormat throws stringThe format a name gives: BITS for keys and values, or KEYS/VALUES ("4/2", "nv4/4"); a code is 16, 8, 4 or 2 bits or nv4 (NVFP4; values only with NV4 keys); "" is the engine's choice (keys 0).
Effects: fail, alloc.
kvName
function kvName(f: KvFormat): stringThe name of a format, as kvFormat reads it.
Effects: fail, alloc.
kvFitted
function kvFitted(a: Arch, f: KvFormat): KvFormatFormat f as the heads of a take it (kvWidth).
Effects: fail.
kvLayout
function kvLayout(a: Arch, layers: i64, f: KvFormat): KvLayoutThe layout of a page of layers pool layers of a's head shape in format f.
Effects: fail.
kvPlan
function kvPlan(a: Arch, wides: Vec<bool>, formats: Vec<KvFormat>): KvPlanThe plan of pool layers of head shapes wides (true: Arch.wideView's) in formats formats, one a pool layer.
Effects: fail, alloc.
kvFormatsOf
function kvFormatsOf(name: string, layers: i64): Vec<KvFormat> throws stringThe formats of layers pool layers that a name gives: kvFormat's for all, or one a pool layer, separated by
commas (as kvAuto writes them).
Effects: fail, alloc.
kvCostsText
function kvCostsText(costs: Vec<Vec<f64>>): stringThe text of measured costs (a model folder's kv-costs, Model.kvCosts): a line of kvFormats' names, then a line of costs an attention layer, in that order.
Effects: fail, alloc.
kvCostsOf
function kvCostsOf(text: string): Vec<Vec<f64>> throws stringThe costs that kvCostsText wrote; the formats must be kvFormats'.
Effects: fail, alloc.
kvPlanName
function kvPlanName(p: KvPlan): stringThe name of a plan's formats as kvFormatsOf reads it: one name when all pool layers take one format.
Effects: fail, alloc.
kvBits
function kvBits(p: KvPlan): f64The mean bits of a pool layer's key or value in plan p.
Effects: fail.
Constants
PAGE
const PAGE: i64 = 16Positions per GPU KV page. All layout and scheduler users share this constant.