# gizai:llm/model a converted model: its linears in their formats, its files, and its load on one or more devices. ```gizlang import * as Model from "gizai:llm/model"; ``` ## Types ### `Linear` ```gizlang type Linear = { n: i64; k: i64; format: i64; codes: bytes; scales: bytes; book: Vec; } ``` A linear in its format (see Formats): `n` rows (a multiple of 16), `k` inputs (a multiple of 128); scales and book empty where the format has none (on a GPU: the codes again). - `n`: `i64` - `k`: `i64` - `format`: `i64` - `codes`: `bytes` - `scales`: `bytes` - `book`: `Vec` ### `DeviceLinear` ```gizlang type DeviceLinear = { n: i64; k: i64; format: i64; codes: Buffer; scales: Buffer; book: Buffer; } ``` On a GPU, a K format's book holds its subs' bytes (see K formats). - `n`: `i64` - `k`: `i64` - `format`: `i64` - `codes`: `M10.Buffer` - `scales`: `M10.Buffer` - `book`: `M10.Buffer` ### `Layer` ```gizlang type Layer = { kind: i64; at: i64; pool: i64; wide: bool; source: i64; layerInput: Vec; window: i64; rope: i64; n1: i64; n2: i64; n3: i64; n4: i64; scalar: i64; qkv: i64; rest: i64; o: i64; qn: i64; kn: i64; bias: i64; input: i64; out: i64; conv: i64; decay: i64; dt: i64; gn: i64; up: i64; down: i64; ah: Vec; mh: Vec; router: i64; ple: Vec; } ``` A layer's tensors: indexes into the model's linears (qkv, o, input, out, up, down) and small tensors (the others), -1 where the layer has none. `at` is its index among the layers of its kind, `pool` among the attention layers of its head shape in the KV pages (`wide`: Arch.wideAt). - `kind`: `i64` - `at`: `i64` - `pool`: `i64` - `wide`: `bool` - `source`: `i64` - `layerInput`: `Vec` - `window`: `i64` - `rope`: `i64` - `n1`: `i64` - `n2`: `i64` - `n3`: `i64` - `n4`: `i64` - `scalar`: `i64` - `qkv`: `i64` - `rest`: `i64` - `o`: `i64` - `qn`: `i64` - `kn`: `i64` - `bias`: `i64` - `input`: `i64` - `out`: `i64` - `conv`: `i64` - `decay`: `i64` - `dt`: `i64` - `gn`: `i64` - `up`: `i64` - `down`: `i64` - `ah`: `Vec` - `mh`: `Vec` - `router`: `i64` - `ple`: `Vec` ### `FlashFiles` ```gizlang type FlashFiles = { fd: i64; at: Vec; record: i64; bits: i64; books: Vec; rows: Vec; mult: Vec; sizes: Vec; offsets: Vec; } ``` A Flash model's own files: the experts' (its descriptor, the file offset of each layer's records, a record's bytes and its two codebooks on the GPU, gate and up, then down) and the PLE's n-gram rows (BF16; none without a PLE) with their hash multipliers, each head's vocabulary size and first row. - `fd`: `i64` - `at`: `Vec` - `record`: `i64` - `bits`: `i64` - `books`: `Vec` - `rows`: `Vec` - `mult`: `Vec` - `sizes`: `Vec` - `offsets`: `Vec` ### `Flash` ```gizlang type Flash = { bufs: Vec; chunks: Vec; perChunk: i64; slotOf: Vec; owner: Vec; used: Vec; busy: Vec; hand: i64; stamp: i64; hits: i64; misses: i64; readNs: i64; history: Vec; convs: Vec>; } ``` Flash's state on a GPU (Engine.flash): its work buffers (F_*), the expert cache and the PLE's states. The cache holds an expert in each slot (expertBytes) in blocks (`chunks`) of `perChunk` slots: slots 0 to layers - 1 the layers' shared experts for good, the others the experts the routers picked last (CLOCK: a slot read since the hand passed it last stays once more). Per expert (layer (experts + 1) + index) its slot or -1; per slot its expert or -1, whether it was read and the stamp of the layer that reads it (not evicted then). The hits and misses of the cache and the ns its reads took. Per slot of the engine the last two tokens (the latest first) and the PLE convolution's last inputs. - `bufs`: `Vec` - `chunks`: `Vec` - `perChunk`: `i64` - `slotOf`: `Vec` - `owner`: `Vec` - `used`: `Vec` - `busy`: `Vec` - `hand`: `i64` - `stamp`: `i64` - `hits`: `i64` - `misses`: `i64` - `readNs`: `i64` - `history`: `Vec` - `convs`: `Vec>` ### `Model` ```gizlang type Model = { arch: Arch; folder: string; tokenizer: Tokenizer; stops: Vec; template: string; generationSuffix?: string; system: string; think: bool; calls: bool; efforts: Vec; sampling: Sampling; embed: Stored; layers: Vec; norm: i64; head: i64; mtp: Option; mixer: Vec; linears: Vec; small: Vec>; gpu: Option; device: Vec; deviceSmall: Vec; signs: Vec; deviceEmbed?: Buffer; layerInput?: { embed: Stored; projection: i64; norm: i64; }; files: FlashFiles; vision?: Vision; stream?: LayerStream; stages: Vec; npu?: NpuLlm.Model; } ``` A converted model, on the GPU that gizhw:gpu opens (GIZLANG_DEVICE) when there is one (`device`), else on the host (`linears`); the small tensors are on both. The embedding stays in the file, and on the GPU of the first layer (deviceEmbed): a step there gathers its tokens' rows on the GPU, the host reads them from the file. `template` is the chat layout ("chatml", "llama3" or ""), `system` the system turn it writes when a chat has none, `think` whether its assistant turn opens with a thinking block, `calls` whether it writes tool calls as ...; `efforts` the instructions its template gives a thinking turn at low, medium and high reasoning effort ("": none); `stops` are the tokens that end a generation and `sampling` the model's own settings. `mtp` is the MTP layer on a GPU when the model has one, `vision` its vision tower when its folder has one (convertVision), `stages` the models of its later layers on other GPUs (loadAcross). - `arch`: `_gizlang_lib_ai__llm__shape.Arch` - `folder`: `string` - `tokenizer`: `M27.Tokenizer` - `stops`: `Vec` - `template`: `string` - `generationSuffix`: `_gizlang_sdk_core.Option` - `system`: `string` - `think`: `bool` - `calls`: `bool` - `efforts`: `Vec` - `sampling`: `M53.Sampling` - `embed`: `gizlang_safetensors.Stored` - `layers`: `Vec` - `norm`: `i64` - `head`: `i64` - `mtp`: `_gizlang_sdk_core.Option` - `mixer`: `Vec` - `linears`: `Vec` - `small`: `Vec>` - `gpu`: `_gizlang_sdk_core.Option` - `device`: `Vec` - `deviceSmall`: `Vec` - `signs`: `Vec` - `deviceEmbed`: `_gizlang_sdk_core.Option` - `layerInput`: `_gizlang_sdk_core.Option<{ embed: gizlang_safetensors.Stored; projection: i64; norm: i64; }>` - `files`: `M51.FlashFiles` - `vision`: `_gizlang_sdk_core.Option<_gizlang_lib_ai__llm__vision.Vision>` - `stream`: `_gizlang_sdk_core.Option` - `stages`: `Vec` - `npu`: `_gizlang_sdk_core.Option<_gizlang_lib_ai__llm__npu.Model>` ### `LayerStream` ```gizlang type LayerStream = { host: Pinned; bytes: i64; slots: Vec; } ``` The layers' linears of a model loaded `layered` (loadOn): layer i's bytes at i × `bytes` of `host`, run from slot i % 2 of the GPU's two `slots`, where every layer's linears have the same places. - `host`: `M10.Pinned` - `bytes`: `i64` - `slots`: `Vec` ### `ChatTokens` ```gizlang type ChatTokens = { tokens: Vec; mark: i64; head: i64; } ``` A chat's prompt as tokens (Model.chatPrompt), the mark of Engine.add (where its messages end, before the assistant's turn starts: the chat's next request starts with these messages, so its slot reads only the rest) and its head (where the system turn with the tools ends: the chats with the same system turn share its keys and values). - `tokens`: `Vec` - `mark`: `i64` - `head`: `i64` ## Functions ### `noFiles` ```gizlang function noFiles(): FlashFiles ``` FlashFiles of no file. Effects: fail. ### `noFlash` ```gizlang function noFlash(): Flash ``` The Flash state of an engine without Flash. ### `load` ```gizlang function load(sys: System, folder: string, onGpu: bool = true, layers: i64 = 0): Model throws string ``` The converted model in `folder`, on the GPU when `onGpu` and there is one, and across the driver's other GPUs too when its weights do not leave SPLIT_ROOM free on that one (loadAcross). With `layers` less than the model's, only its first `layers` layers, without the final norm, the head and the MTP layer: an engine of it gives hidden states (Engine.hidden), not logits. A gizai:llm/npu folder (model.q4nx) runs on this host's NPU (Model.npu). Effects: fail, read, write, console, io. ### `loadAcross` ```gizlang function loadAcross(sys: System, folder: string, gpus: Vec, counts: Vec = []): Model throws string ``` The converted model in `folder` across `gpus` (Gpu.others), in stages of consecutive layers: the first GPU holds the first stage (and the vision tower), the last the last stage with the final norm and the head (Model.stages). A stage's layer count is counts[i], by default in proportion to its GPU's free memory. A step runs the stages in turn, each on its rows' hidden states after the one before (Engine). Without the MTP layer, and not for Qwen3.8 Flash yet. The first GPU is current at the end. Effects: fail, read, write, console, io. ### `loadOn` ```gizlang function loadOn(sys: System, folder: string, gpu: Option, layers: i64 = 0, from: i64 = 0, streamed: bool = false, layered: bool = false): Result ``` `load` on the GPU `gpu` (none: the host). With `from`, the `layers` layers from layer `from` on (a stage of loadAcross): the norm, the head and the MTP layer only where they are the model's last. `streamed`: the vision tower's linears stay in host memory (loadVision). `layered`: the layers' linears stay in page-locked host memory and each pass copies them to two layers' room on the GPU, a layer ahead of its use (Model.stream). Effects: fail, read, write, console, io. ### `converted` ```gizlang function converted(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string throws string ``` The folder of the model in folder `folder` ("" for its top) of Hugging Face repository `repo` ("Qwen/Qwen3.5-4B") converted with `bits` (and `layers`, as `convert` cuts), for `load` with the same `layers`: the folder is downloaded through `hub` once (Hub.snapshot) and converted once into `.gizlang-llm/REPO[/FOLDER]/BITS` in the hub's folder. A conversion is written to a partial folder and renamed when it is complete, so a stopped one is done again, never loaded. Effects: fail, read, write, console, io. ### `convertedFolder` ```gizlang function convertedFolder(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string ``` Where `converted` puts a conversion; the folder holds model.safetensors once the conversion is complete. Effects: fail. ### `toolCalls` ```gizlang function toolCalls(text: string, tools: Vec, first: i64 = 0): Vec ``` The tool calls in the text a model wrote after its answer (Model.calls: VALUE...), each value as JSON by the type its tool gives the parameter: a string stays text, another type is read as JSON (text when it is not). Their ids are call_FIRST_K. Effects: fail. ## Methods ### `Model.chatPrompt` ```gizlang function Model.chatPrompt(self: Model, messages: Vec, thinking: bool, tools: Vec = [], effort: string = "high"): string ``` The prompt of a chat as the model's template writes it, ending with the start of the assistant's turn (with an empty thinking block unless `thinking`), a thinking turn with the instruction of its reasoning `effort` (low, medium or high: Model.efforts), and the tools it may call (OpenAI's tool objects, for a model whose template writes tool calls: Model.calls). Effects: fail. ### `Model.chatTokens` ```gizlang function Model.chatTokens(self: Model, messages: Vec, thinking: bool, tools: Vec = [], effort: string = "high"): ChatTokens ``` The tokens of `messages` in the model's chat template, up to where the assistant's turn starts (ChatTokens). Effects: fail. ### `Model.engine` ```gizlang function Model.engine(self: Model, slots: i64, context: i64, reserve: i64 = 134_217_728, drafts: i64 = -1, rows: i64 = 0, kv: string = ""): Result ``` An engine for `slots` sequences of up to `context` positions, reading up to `rows` rows a step (0: 2048 on a GPU, as vLLM's batched tokens, else 256; at least 1 + drafts rows a slot, 2 without drafts). On a GPU the KV pages take the device memory left after the work buffers and recurrent states, at most what the slots can hold, keeping `reserve` bytes free. Keys and values take the format `kv` names (kvFormat; F16 where the heads take no codes, see kvWidth), by default the first of kvFormats in which the slots' whole contexts fit above 4 bits, or one whole context at most 4 bits, else the last (on the host 8 bits). An engine reads up to `drafts` drafts a sequence and step (0: none; 3 is DRAFTS), and with the model's MTP layer Engine.advance drafts them where that gives tokens faster (EXPLORE), on a GPU or the host (where 1 draft is the fastest: its 4-bit layers take two rows in the time of one, and each further draft costs a step of the MTP layer). Effects: fail, read, write, console, io. ## Constants ### `CALL` ```gizlang const CALL = "" ``` Where a model that writes tool calls (Model.calls) starts them.