GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/llm/kvformat

One owner for KV page packing and quantization layout, independent of execution and scheduling.

gizlang
import * as Kvformat from "gizai/llm/kvformat";

Types

KvFormat

gizlang
type KvFormat = { keys: i64; values: i64; }

How an engine keeps keys and values: the code of a key and of a value. 16: F16 values; 8, 4 or 2: codes of the head rotated by blocks of kvBlock, at 2^bits even steps (kvStore); NV4 (NVFP4): E2M1 codes of the rotated head, which the FP4 matrix cores take with no decoding. NV4 keys have an E4M3 scale a block of 16 dims and an F32 scale a head; NV4 values (with NV4 keys only) an F32 scale a head, their codes kept by dims (KvPart.columns).

  • keys: i64
  • values: i64

KvPart

gizlang
type KvPart = { bits: i64; word: i64; each: i64; at: i64; page: i64; steps: i64; floats: i64; nv: bool; scales: i64; columns: bool; }

Where a KV page keeps one kind (keys or values) of its heads, each pool layer, kv head and position (kvHead): F16 values, or 32 lanes of codes, lane l the codes of values l hd/32 on; each words of type word a lane, from word at of the page (page words of that type). With codes each head's step and least value (F32) from float steps. With nv (NV4) a head's F32 scale is its first float there. NV4 keys: the E4M3 block scales hd/64 words (4 a word) a head from float scales, and a head's 16-byte chunk c at chunk c ^ (position % (hd / 32)) of its row (kvLane), so the rows a matrix load reads together meet distinct banks. NV4 values (columns): a page's PAGE heads of a pool layer and kv head by dims, dim d in words 2 d and 2 d + 1 (I32), position p's code in nibble p % 8 of word p / 8: the B tiles of the FP4 products with the weights (flashNvKernel).

  • bits: i64
  • word: i64
  • each: i64
  • at: i64
  • page: i64
  • steps: i64
  • floats: i64
  • nv: bool
  • scales: i64
  • columns: bool

KvLayout

gizlang
type KvLayout = { keys: KvPart; values: KvPart; bytes: i64; }

A page's keys and values of the layers pool layers of a's head shape (KvPart): the keys' heads, the values' heads, their steps, then the block scales of NV4 keys; bytes their size. They start at byte Arch.kvFrom of a shared page (kvViews).

  • keys: M65.KvPart
  • values: M65.KvPart
  • bytes: i64

KvGroup

gizlang
type KvGroup = { arch: Arch; format: KvFormat; layers: i64; }

The pool layers of one head shape and one format: their Arch with their region of a KV page (Arch.kvFrom, kvPage) and how many they are.

  • arch: _gizlang_lib_ai__llm__shape.Arch
  • format: M65.KvFormat
  • layers: i64

KvPlace

gizlang
type KvPlace = { group: i64; index: i64; }

A pool layer's group and its index among the group's layers.

  • group: i64
  • index: i64

KvPlan

gizlang
type KvPlan = { formats: Vec<KvFormat>; groups: Vec<KvGroup>; places: Vec<KvPlace>; bytes: i64; }

How a KV page keeps each pool layer: its format (fitted, kvFitted), its group and its index there; the groups' regions one after the other, bytes the page. One group (one shape, one format) takes the whole page.

  • formats: Vec<M65.KvFormat>
  • groups: Vec<M65.KvGroup>
  • places: Vec<M65.KvPlace>
  • bytes: i64

Functions

kvFormats

gizlang
function kvFormats(): Vec<KvFormat>

The formats an engine chooses from for each pool layer (kvChoice), most bits first. A key's error moves all scores of its position, a value's only its share of the sum: values lose bits first.

Effects: fail, alloc.

kvPrior

gizlang
function kvPrior(): Vec<f64>

A pool layer's cost in each of kvFormats where its model has no measured costs (kvCosts): the KL of Qwen3.8-27B's next tokens with all its 16 pool layers in the format, a 16th each; nv4 is a guess between nv4/4 and 4 until it is measured.

Effects: fail, alloc.

kvCodeBits

gizlang
function kvCodeBits(code: i64): i64

The bits a code takes for each value (NV4: 4, its scales apart).

kvAllowed

gizlang
function kvAllowed(codes: string): Vec<bool> throws string

Which kinds of codes a format list keeps (an engine's kvCodes): "all", "nv4" (no 4-bit keys at even steps: the FP4 matrix cores take every 4-bit key) or "uniform" (no NV4).

Effects: fail, alloc.

kvChoice

gizlang
function kvChoice(a: Arch, wides: Vec<bool>, costs: Vec<Vec<f64>>, room: i64, allowed: Vec<bool>): Vec<KvFormat>

The formats of the pool layers (head shapes wides, as kvPlan) of least summed cost (costs: a row a pool layer, a cost for each of kvFormats) in which a page takes at most room bytes, each from the kvFormats that allowed marks; [] where none fits. An exact knapsack: the costs of each layer's change add up (as AutoQuant KV takes them).

Effects: fail, alloc.

kvFormat

gizlang
function kvFormat(name: string): KvFormat throws string

The format a name gives: BITS for keys and values, or KEYS/VALUES ("4/2", "nv4/4"); a code is 16, 8, 4 or 2 bits or nv4 (NVFP4; values only with NV4 keys); "" is the engine's choice (keys 0).

Effects: fail, alloc.

kvName

gizlang
function kvName(f: KvFormat): string

The name of a format, as kvFormat reads it.

Effects: fail, alloc.

kvFitted

gizlang
function kvFitted(a: Arch, f: KvFormat): KvFormat

Format f as the heads of a take it (kvWidth).

Effects: fail.

kvLayout

gizlang
function kvLayout(a: Arch, layers: i64, f: KvFormat): KvLayout

The layout of a page of layers pool layers of a's head shape in format f.

Effects: fail.

kvPlan

gizlang
function kvPlan(a: Arch, wides: Vec<bool>, formats: Vec<KvFormat>): KvPlan

The plan of pool layers of head shapes wides (true: Arch.wideView's) in formats formats, one a pool layer.

Effects: fail, alloc.

kvFormatsOf

gizlang
function kvFormatsOf(name: string, layers: i64): Vec<KvFormat> throws string

The formats of layers pool layers that a name gives: kvFormat's for all, or one a pool layer, separated by commas (as kvAuto writes them).

Effects: fail, alloc.

kvCostsText

gizlang
function kvCostsText(costs: Vec<Vec<f64>>): string

The text of measured costs (a model folder's kv-costs, Model.kvCosts): a line of kvFormats' names, then a line of costs an attention layer, in that order.

Effects: fail, alloc.

kvCostsOf

gizlang
function kvCostsOf(text: string): Vec<Vec<f64>> throws string

The costs that kvCostsText wrote; the formats must be kvFormats'.

Effects: fail, alloc.

kvPlanName

gizlang
function kvPlanName(p: KvPlan): string

The name of a plan's formats as kvFormatsOf reads it: one name when all pool layers take one format.

Effects: fail, alloc.

kvBits

gizlang
function kvBits(p: KvPlan): f64

The mean bits of a pool layer's key or value in plan p.

Effects: fail.

Constants

PAGE

gizlang
const PAGE: i64 = 16

Positions per GPU KV page. All layout and scheduler users share this constant.