# gizai:llm/model

a converted model: its linears in their formats, its files, and its load on one or more devices.

```gizlang
import * as Model from "gizai:llm/model";
```

## Types

### `Linear`

```gizlang
type Linear = { n: i64; k: i64; format: i64; codes: bytes; scales: bytes; book: Vec<f32>; }
```

A linear in its format (see Formats): `n` rows (a multiple of 16), `k` inputs (a multiple of 128); scales and book
empty where the format has none (on a GPU: the codes again).

- `n`: `i64`
- `k`: `i64`
- `format`: `i64`
- `codes`: `bytes`
- `scales`: `bytes`
- `book`: `Vec<f32>`

### `DeviceLinear`

```gizlang
type DeviceLinear = { n: i64; k: i64; format: i64; codes: Buffer; scales: Buffer; book: Buffer; }
```

On a GPU, a K format's book holds its subs' bytes (see K formats).

- `n`: `i64`
- `k`: `i64`
- `format`: `i64`
- `codes`: `M10.Buffer`
- `scales`: `M10.Buffer`
- `book`: `M10.Buffer`

### `Layer`

```gizlang
type Layer = { kind: i64; at: i64; pool: i64; wide: bool; source: i64; layerInput: Vec<i64>; window: i64; rope: i64; n1: i64; n2: i64; n3: i64; n4: i64; scalar: i64; qkv: i64; rest: i64; o: i64; qn: i64; kn: i64; bias: i64; input: i64; out: i64; conv: i64; decay: i64; dt: i64; gn: i64; up: i64; down: i64; ah: Vec<i64>; mh: Vec<i64>; router: i64; ple: Vec<i64>; }
```

A layer's tensors: indexes into the model's linears (qkv, o, input, out, up, down) and small tensors (the
others), -1 where the layer has none. `at` is its index among the layers of its kind, `pool` among the attention
layers of its head shape in the KV pages (`wide`: Arch.wideAt).

- `kind`: `i64`
- `at`: `i64`
- `pool`: `i64`
- `wide`: `bool`
- `source`: `i64`
- `layerInput`: `Vec<i64>`
- `window`: `i64`
- `rope`: `i64`
- `n1`: `i64`
- `n2`: `i64`
- `n3`: `i64`
- `n4`: `i64`
- `scalar`: `i64`
- `qkv`: `i64`
- `rest`: `i64`
- `o`: `i64`
- `qn`: `i64`
- `kn`: `i64`
- `bias`: `i64`
- `input`: `i64`
- `out`: `i64`
- `conv`: `i64`
- `decay`: `i64`
- `dt`: `i64`
- `gn`: `i64`
- `up`: `i64`
- `down`: `i64`
- `ah`: `Vec<i64>`
- `mh`: `Vec<i64>`
- `router`: `i64`
- `ple`: `Vec<i64>`

### `FlashFiles`

```gizlang
type FlashFiles = { fd: i64; at: Vec<i64>; record: i64; bits: i64; books: Vec<Buffer>; rows: Vec<Stored>; mult: Vec<u64>; sizes: Vec<i64>; offsets: Vec<i64>; }
```

A Flash model's own files: the experts' (its descriptor, the file offset of each layer's records, a record's
bytes and its two codebooks on the GPU, gate and up, then down) and the PLE's n-gram rows (BF16; none without a
PLE) with their hash multipliers, each head's vocabulary size and first row.

- `fd`: `i64`
- `at`: `Vec<i64>`
- `record`: `i64`
- `bits`: `i64`
- `books`: `Vec<M10.Buffer>`
- `rows`: `Vec<gizlang_safetensors.Stored>`
- `mult`: `Vec<u64>`
- `sizes`: `Vec<i64>`
- `offsets`: `Vec<i64>`

### `Flash`

```gizlang
type Flash = { bufs: Vec<Buffer>; chunks: Vec<Buffer>; perChunk: i64; slotOf: Vec<i64>; owner: Vec<i64>; used: Vec<bool>; busy: Vec<i64>; hand: i64; stamp: i64; hits: i64; misses: i64; readNs: i64; history: Vec<i64>; convs: Vec<Vec<f32>>; }
```

Flash's state on a GPU (Engine.flash): its work buffers (F_*), the expert cache and the PLE's states. The cache
holds an expert in each slot (expertBytes) in blocks (`chunks`) of `perChunk` slots: slots 0 to layers - 1 the
layers' shared experts for good, the others the experts the routers picked last (CLOCK: a slot read since the hand
passed it last stays once more). Per expert (layer (experts + 1) + index) its slot or -1; per slot its expert or -1,
whether it was read and the stamp of the layer that reads it (not evicted then). The hits and misses of the cache
and the ns its reads took. Per slot of the engine the last two tokens (the latest first) and the PLE convolution's
last inputs.

- `bufs`: `Vec<M10.Buffer>`
- `chunks`: `Vec<M10.Buffer>`
- `perChunk`: `i64`
- `slotOf`: `Vec<i64>`
- `owner`: `Vec<i64>`
- `used`: `Vec<bool>`
- `busy`: `Vec<i64>`
- `hand`: `i64`
- `stamp`: `i64`
- `hits`: `i64`
- `misses`: `i64`
- `readNs`: `i64`
- `history`: `Vec<i64>`
- `convs`: `Vec<Vec<f32>>`

### `Model`

```gizlang
type Model = { arch: Arch; folder: string; tokenizer: Tokenizer; stops: Vec<i64>; template: string; generationSuffix?: string; system: string; think: bool; calls: bool; efforts: Vec<string>; sampling: Sampling; embed: Stored; layers: Vec<Layer>; norm: i64; head: i64; mtp: Option<Mtp>; mixer: Vec<i64>; linears: Vec<Linear>; small: Vec<Vec<f32>>; gpu: Option<Gpu>; device: Vec<DeviceLinear>; deviceSmall: Vec<Buffer>; signs: Vec<Buffer>; deviceEmbed?: Buffer; layerInput?: { embed: Stored; projection: i64; norm: i64; }; files: FlashFiles; vision?: Vision; stream?: LayerStream; stages: Vec<Model>; npu?: NpuLlm.Model; }
```

A converted model, on the GPU that gizhw:gpu opens (GIZLANG_DEVICE) when there is one (`device`), else on the
host (`linears`); the small tensors are on both. The embedding stays in the file, and on the GPU of the first layer
(deviceEmbed): a step there gathers its tokens' rows on the GPU, the host reads them from the file.
`template` is the chat layout ("chatml", "llama3" or ""), `system` the system turn it writes when a chat has none,
`think` whether its assistant turn opens with a thinking
block, `calls` whether it writes tool calls as <tool_call><function=NAME><parameter=P>...; `efforts` the instructions
its template gives a thinking turn at low, medium and high reasoning effort ("": none); `stops` are the tokens that end
a generation and `sampling` the model's own settings. `mtp` is the MTP layer on a GPU when the model has one, `vision`
its vision tower when its folder has one (convertVision), `stages` the models of its later layers on other GPUs
(loadAcross).

- `arch`: `_gizlang_lib_ai__llm__shape.Arch`
- `folder`: `string`
- `tokenizer`: `M27.Tokenizer`
- `stops`: `Vec<i64>`
- `template`: `string`
- `generationSuffix`: `_gizlang_sdk_core.Option<string>`
- `system`: `string`
- `think`: `bool`
- `calls`: `bool`
- `efforts`: `Vec<string>`
- `sampling`: `M53.Sampling`
- `embed`: `gizlang_safetensors.Stored`
- `layers`: `Vec<M51.Layer>`
- `norm`: `i64`
- `head`: `i64`
- `mtp`: `_gizlang_sdk_core.Option<M51.Mtp>`
- `mixer`: `Vec<i64>`
- `linears`: `Vec<M51.Linear>`
- `small`: `Vec<Vec<f32>>`
- `gpu`: `_gizlang_sdk_core.Option<M10.Gpu>`
- `device`: `Vec<M51.DeviceLinear>`
- `deviceSmall`: `Vec<M10.Buffer>`
- `signs`: `Vec<M10.Buffer>`
- `deviceEmbed`: `_gizlang_sdk_core.Option<M10.Buffer>`
- `layerInput`: `_gizlang_sdk_core.Option<{ embed: gizlang_safetensors.Stored; projection: i64; norm: i64; }>`
- `files`: `M51.FlashFiles`
- `vision`: `_gizlang_sdk_core.Option<_gizlang_lib_ai__llm__vision.Vision>`
- `stream`: `_gizlang_sdk_core.Option<M51.LayerStream>`
- `stages`: `Vec<M51.Model>`
- `npu`: `_gizlang_sdk_core.Option<_gizlang_lib_ai__llm__npu.Model>`

### `LayerStream`

```gizlang
type LayerStream = { host: Pinned; bytes: i64; slots: Vec<Buffer>; }
```

The layers' linears of a model loaded `layered` (loadOn): layer i's bytes at i × `bytes` of `host`, run from slot i % 2
of the GPU's two `slots`, where every layer's linears have the same places.

- `host`: `M10.Pinned`
- `bytes`: `i64`
- `slots`: `Vec<M10.Buffer>`

### `ChatTokens`

```gizlang
type ChatTokens = { tokens: Vec<i64>; mark: i64; head: i64; }
```

A chat's prompt as tokens (Model.chatPrompt), the mark of Engine.add (where its messages end, before the assistant's
turn starts: the chat's next request starts with these messages, so its slot reads only the rest) and its head (where
the system turn with the tools ends: the chats with the same system turn share its keys and values).

- `tokens`: `Vec<i64>`
- `mark`: `i64`
- `head`: `i64`

## Functions

### `noFiles`

```gizlang
function noFiles(): FlashFiles
```

FlashFiles of no file.

Effects: fail.

### `noFlash`

```gizlang
function noFlash(): Flash
```

The Flash state of an engine without Flash.

### `load`

```gizlang
function load(sys: System, folder: string, onGpu: bool = true, layers: i64 = 0): Model throws string
```

The converted model in `folder`, on the GPU when `onGpu` and there is one, and across the driver's other GPUs too
when its weights do not leave SPLIT_ROOM free on that one (loadAcross). With `layers` less than the model's, only its
first `layers` layers, without the final norm, the head and the MTP layer: an engine of it gives hidden states
(Engine.hidden), not logits. A gizai:llm/npu folder (model.q4nx) runs on this host's NPU (Model.npu).

Effects: fail, read, write, console, io.

### `loadAcross`

```gizlang
function loadAcross(sys: System, folder: string, gpus: Vec<Gpu>, counts: Vec<i64> = []): Model throws string
```

The converted model in `folder` across `gpus` (Gpu.others), in stages of consecutive layers: the first GPU holds the
first stage (and the vision tower), the last the last stage with the final norm and the head (Model.stages). A
stage's layer count is counts[i], by default in proportion to its GPU's free memory. A step runs the stages in turn,
each on its rows' hidden states after the one before (Engine). Without the MTP layer, and not for Qwen3.8 Flash
yet. The first GPU is current at the end.

Effects: fail, read, write, console, io.

### `loadOn`

```gizlang
function loadOn(sys: System, folder: string, gpu: Option<Gpu>, layers: i64 = 0, from: i64 = 0, streamed: bool = false, layered: bool = false): Result<Model, string>
```

`load` on the GPU `gpu` (none: the host). With `from`, the `layers` layers from layer `from` on (a stage of
loadAcross): the norm, the head and the MTP layer only where they are the model's last. `streamed`: the vision
tower's linears stay in host memory (loadVision). `layered`: the layers' linears stay in page-locked host memory
and each pass copies them to two layers' room on the GPU, a layer ahead of its use (Model.stream).

Effects: fail, read, write, console, io.

### `converted`

```gizlang
function converted(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string throws string
```

The folder of the model in folder `folder` ("" for its top) of Hugging Face repository `repo` ("Qwen/Qwen3.5-4B")
converted with `bits` (and `layers`, as `convert` cuts), for `load` with the same `layers`: the folder is downloaded
through `hub` once (Hub.snapshot) and converted once into `.gizlang-llm/REPO[/FOLDER]/BITS` in the hub's folder. A
conversion is written to a partial folder and renamed when it is complete, so a stopped one is done again, never
loaded.

Effects: fail, read, write, console, io.

### `convertedFolder`

```gizlang
function convertedFolder(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string
```

Where `converted` puts a conversion; the folder holds model.safetensors once the conversion is complete.

Effects: fail.

### `toolCalls`

```gizlang
function toolCalls(text: string, tools: Vec<JsonValue>, first: i64 = 0): Vec<OpenAi.Call>
```

The tool calls in the text a model wrote after its answer (Model.calls: <tool_call><function=NAME><parameter=P>
VALUE</parameter>...), each value as JSON by the type its tool gives the parameter: a string stays text, another
type is read as JSON (text when it is not). Their ids are call_FIRST_K.

Effects: fail.

## Methods

### `Model.chatPrompt`

```gizlang
function Model.chatPrompt(self: Model, messages: Vec<Message>, thinking: bool, tools: Vec<JsonValue> = [], effort: string = "high"): string
```

The prompt of a chat as the model's template writes it, ending with the start of the assistant's turn (with an
empty thinking block unless `thinking`), a thinking turn with the instruction of its reasoning `effort` (low, medium or
high: Model.efforts), and the tools it may call (OpenAI's tool objects, for a model whose
template writes tool calls: Model.calls).

Effects: fail.

### `Model.chatTokens`

```gizlang
function Model.chatTokens(self: Model, messages: Vec<Message>, thinking: bool, tools: Vec<JsonValue> = [], effort: string = "high"): ChatTokens
```

The tokens of `messages` in the model's chat template, up to where the assistant's turn starts (ChatTokens).

Effects: fail.

### `Model.engine`

```gizlang
function Model.engine(self: Model, slots: i64, context: i64, reserve: i64 = 134_217_728, drafts: i64 = -1, rows: i64 = 0, kv: string = ""): Result<Engine, string>
```

An engine for `slots` sequences of up to `context` positions, reading up to `rows` rows a step (0: 2048 on a GPU, as
vLLM's batched tokens, else 256; at least
1 + drafts rows a slot, 2 without drafts). On a GPU the KV pages take the device memory left after the work buffers
and recurrent states, at most what the slots can hold, keeping `reserve` bytes free. Keys and values take the format
`kv` names (kvFormat; F16 where the heads take no codes, see kvWidth), by default the first of kvFormats in which
the slots' whole contexts fit above 4 bits, or one whole context at most 4 bits, else the last (on the host 8 bits). An engine reads up to `drafts` drafts a sequence and step (0: none; 3 is
DRAFTS), and with the model's MTP layer Engine.advance drafts them where that gives tokens faster (EXPLORE), on a GPU
or the host (where 1 draft is the fastest: its 4-bit layers take two rows in the time of one, and each further draft
costs a step of the MTP layer).

Effects: fail, read, write, console, io.

## Constants

### `CALL`

```gizlang
const CALL = "<tool_call>"
```

Where a model that writes tool calls (Model.calls) starts them.

gizai:llm/model

a converted model: its linears in their formats, its files, and its load on one or more devices.

gizlang

import * as Model from "gizai:llm/model";

Types

`Linear`

gizlang

type Linear = { n: i64; k: i64; format: i64; codes: bytes; scales: bytes; book: Vec<f32>; }

A linear in its format (see Formats): n rows (a multiple of 16), k inputs (a multiple of 128); scales and book empty where the format has none (on a GPU: the codes again).

n: i64
k: i64
format: i64
codes: bytes
scales: bytes
book: Vec<f32>

`DeviceLinear`

gizlang

type DeviceLinear = { n: i64; k: i64; format: i64; codes: Buffer; scales: Buffer; book: Buffer; }

On a GPU, a K format's book holds its subs' bytes (see K formats).

n: i64
k: i64
format: i64
codes: M10.Buffer
scales: M10.Buffer
book: M10.Buffer

`Layer`

gizlang

type Layer = { kind: i64; at: i64; pool: i64; wide: bool; source: i64; layerInput: Vec<i64>; window: i64; rope: i64; n1: i64; n2: i64; n3: i64; n4: i64; scalar: i64; qkv: i64; rest: i64; o: i64; qn: i64; kn: i64; bias: i64; input: i64; out: i64; conv: i64; decay: i64; dt: i64; gn: i64; up: i64; down: i64; ah: Vec<i64>; mh: Vec<i64>; router: i64; ple: Vec<i64>; }

A layer's tensors: indexes into the model's linears (qkv, o, input, out, up, down) and small tensors (the others), -1 where the layer has none. at is its index among the layers of its kind, pool among the attention layers of its head shape in the KV pages (wide: Arch.wideAt).

kind: i64
at: i64
pool: i64
wide: bool
source: i64
layerInput: Vec<i64>
window: i64
rope: i64
n1: i64
n2: i64
n3: i64
n4: i64
scalar: i64
qkv: i64
rest: i64
o: i64
qn: i64
kn: i64
bias: i64
input: i64
out: i64
conv: i64
decay: i64
dt: i64
gn: i64
up: i64
down: i64
ah: Vec<i64>
mh: Vec<i64>
router: i64
ple: Vec<i64>

`FlashFiles`

gizlang

type FlashFiles = { fd: i64; at: Vec<i64>; record: i64; bits: i64; books: Vec<Buffer>; rows: Vec<Stored>; mult: Vec<u64>; sizes: Vec<i64>; offsets: Vec<i64>; }

A Flash model's own files: the experts' (its descriptor, the file offset of each layer's records, a record's bytes and its two codebooks on the GPU, gate and up, then down) and the PLE's n-gram rows (BF16; none without a PLE) with their hash multipliers, each head's vocabulary size and first row.

fd: i64
at: Vec<i64>
record: i64
bits: i64
books: Vec<M10.Buffer>
rows: Vec<gizlang_safetensors.Stored>
mult: Vec<u64>
sizes: Vec<i64>
offsets: Vec<i64>

`Flash`

gizlang

type Flash = { bufs: Vec<Buffer>; chunks: Vec<Buffer>; perChunk: i64; slotOf: Vec<i64>; owner: Vec<i64>; used: Vec<bool>; busy: Vec<i64>; hand: i64; stamp: i64; hits: i64; misses: i64; readNs: i64; history: Vec<i64>; convs: Vec<Vec<f32>>; }

Flash's state on a GPU (Engine.flash): its work buffers (F_*), the expert cache and the PLE's states. The cache holds an expert in each slot (expertBytes) in blocks (chunks) of perChunk slots: slots 0 to layers - 1 the layers' shared experts for good, the others the experts the routers picked last (CLOCK: a slot read since the hand passed it last stays once more). Per expert (layer (experts + 1) + index) its slot or -1; per slot its expert or -1, whether it was read and the stamp of the layer that reads it (not evicted then). The hits and misses of the cache and the ns its reads took. Per slot of the engine the last two tokens (the latest first) and the PLE convolution's last inputs.

bufs: Vec<M10.Buffer>
chunks: Vec<M10.Buffer>
perChunk: i64
slotOf: Vec<i64>
owner: Vec<i64>
used: Vec<bool>
busy: Vec<i64>
hand: i64
stamp: i64
hits: i64
misses: i64
readNs: i64
history: Vec<i64>
convs: Vec<Vec<f32>>

`Model`

gizlang

type Model = { arch: Arch; folder: string; tokenizer: Tokenizer; stops: Vec<i64>; template: string; generationSuffix?: string; system: string; think: bool; calls: bool; efforts: Vec<string>; sampling: Sampling; embed: Stored; layers: Vec<Layer>; norm: i64; head: i64; mtp: Option<Mtp>; mixer: Vec<i64>; linears: Vec<Linear>; small: Vec<Vec<f32>>; gpu: Option<Gpu>; device: Vec<DeviceLinear>; deviceSmall: Vec<Buffer>; signs: Vec<Buffer>; deviceEmbed?: Buffer; layerInput?: { embed: Stored; projection: i64; norm: i64; }; files: FlashFiles; vision?: Vision; stream?: LayerStream; stages: Vec<Model>; npu?: NpuLlm.Model; }

A converted model, on the GPU that gizhw:gpu opens (GIZLANG_DEVICE) when there is one (device), else on the host (linears); the small tensors are on both. The embedding stays in the file, and on the GPU of the first layer (deviceEmbed): a step there gathers its tokens' rows on the GPU, the host reads them from the file. template is the chat layout ("chatml", "llama3" or ""), system the system turn it writes when a chat has none, think whether its assistant turn opens with a thinking block, calls whether it writes tool calls as <tool_call><function=NAME><parameter=P>...; efforts the instructions its template gives a thinking turn at low, medium and high reasoning effort ("": none); stops are the tokens that end a generation and sampling the model's own settings. mtp is the MTP layer on a GPU when the model has one, vision its vision tower when its folder has one (convertVision), stages the models of its later layers on other GPUs (loadAcross).

arch: _gizlang_lib_ai__llm__shape.Arch
folder: string
tokenizer: M27.Tokenizer
stops: Vec<i64>
template: string
generationSuffix: _gizlang_sdk_core.Option<string>
system: string
think: bool
calls: bool
efforts: Vec<string>
sampling: M53.Sampling
embed: gizlang_safetensors.Stored
layers: Vec<M51.Layer>
norm: i64
head: i64
mtp: _gizlang_sdk_core.Option<M51.Mtp>
mixer: Vec<i64>
linears: Vec<M51.Linear>
small: Vec<Vec<f32>>
gpu: _gizlang_sdk_core.Option<M10.Gpu>
device: Vec<M51.DeviceLinear>
deviceSmall: Vec<M10.Buffer>
signs: Vec<M10.Buffer>
deviceEmbed: _gizlang_sdk_core.Option<M10.Buffer>
layerInput: _gizlang_sdk_core.Option<{ embed: gizlang_safetensors.Stored; projection: i64; norm: i64; }>
files: M51.FlashFiles
vision: _gizlang_sdk_core.Option<_gizlang_lib_ai__llm__vision.Vision>
stream: _gizlang_sdk_core.Option<M51.LayerStream>
stages: Vec<M51.Model>
npu: _gizlang_sdk_core.Option<_gizlang_lib_ai__llm__npu.Model>

`LayerStream`

gizlang

type LayerStream = { host: Pinned; bytes: i64; slots: Vec<Buffer>; }

The layers' linears of a model loaded layered (loadOn): layer i's bytes at i × bytes of host, run from slot i % 2 of the GPU's two slots, where every layer's linears have the same places.

host: M10.Pinned
bytes: i64
slots: Vec<M10.Buffer>

`ChatTokens`

gizlang

type ChatTokens = { tokens: Vec<i64>; mark: i64; head: i64; }

A chat's prompt as tokens (Model.chatPrompt), the mark of Engine.add (where its messages end, before the assistant's turn starts: the chat's next request starts with these messages, so its slot reads only the rest) and its head (where the system turn with the tools ends: the chats with the same system turn share its keys and values).

tokens: Vec<i64>
mark: i64
head: i64

Functions

`noFiles`

gizlang

function noFiles(): FlashFiles

FlashFiles of no file.

Effects: fail.

`noFlash`

gizlang

function noFlash(): Flash

The Flash state of an engine without Flash.

`load`

gizlang

function load(sys: System, folder: string, onGpu: bool = true, layers: i64 = 0): Model throws string

The converted model in folder, on the GPU when onGpu and there is one, and across the driver's other GPUs too when its weights do not leave SPLIT_ROOM free on that one (loadAcross). With layers less than the model's, only its first layers layers, without the final norm, the head and the MTP layer: an engine of it gives hidden states (Engine.hidden), not logits. A gizai:llm/npu folder (model.q4nx) runs on this host's NPU (Model.npu).

Effects: fail, read, write, console, io.

`loadAcross`

gizlang

function loadAcross(sys: System, folder: string, gpus: Vec<Gpu>, counts: Vec<i64> = []): Model throws string

The converted model in folder across gpus (Gpu.others), in stages of consecutive layers: the first GPU holds the first stage (and the vision tower), the last the last stage with the final norm and the head (Model.stages). A stage's layer count is counts[i], by default in proportion to its GPU's free memory. A step runs the stages in turn, each on its rows' hidden states after the one before (Engine). Without the MTP layer, and not for Qwen3.8 Flash yet. The first GPU is current at the end.

Effects: fail, read, write, console, io.

`loadOn`

gizlang

function loadOn(sys: System, folder: string, gpu: Option<Gpu>, layers: i64 = 0, from: i64 = 0, streamed: bool = false, layered: bool = false): Result<Model, string>

load on the GPU gpu (none: the host). With from, the layers layers from layer from on (a stage of loadAcross): the norm, the head and the MTP layer only where they are the model's last. streamed: the vision tower's linears stay in host memory (loadVision). layered: the layers' linears stay in page-locked host memory and each pass copies them to two layers' room on the GPU, a layer ahead of its use (Model.stream).

Effects: fail, read, write, console, io.

`converted`

gizlang

function converted(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string throws string

The folder of the model in folder folder ("" for its top) of Hugging Face repository repo ("Qwen/Qwen3.5-4B") converted with bits (and layers, as convert cuts), for load with the same layers: the folder is downloaded through hub once (Hub.snapshot) and converted once into .gizlang-llm/REPO[/FOLDER]/BITS in the hub's folder. A conversion is written to a partial folder and renamed when it is complete, so a stopped one is done again, never loaded.

Effects: fail, read, write, console, io.

`convertedFolder`

gizlang

function convertedFolder(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string

Where converted puts a conversion; the folder holds model.safetensors once the conversion is complete.

Effects: fail.

`toolCalls`

gizlang

function toolCalls(text: string, tools: Vec<JsonValue>, first: i64 = 0): Vec<OpenAi.Call>

The tool calls in the text a model wrote after its answer (Model.calls: <tool_call><function=NAME><parameter=P> VALUE</parameter>...), each value as JSON by the type its tool gives the parameter: a string stays text, another type is read as JSON (text when it is not). Their ids are call_FIRST_K.

Effects: fail.

Methods

`Model.chatPrompt`

gizlang

function Model.chatPrompt(self: Model, messages: Vec<Message>, thinking: bool, tools: Vec<JsonValue> = [], effort: string = "high"): string

The prompt of a chat as the model's template writes it, ending with the start of the assistant's turn (with an empty thinking block unless thinking), a thinking turn with the instruction of its reasoning effort (low, medium or high: Model.efforts), and the tools it may call (OpenAI's tool objects, for a model whose template writes tool calls: Model.calls).

Effects: fail.

`Model.chatTokens`

gizlang

function Model.chatTokens(self: Model, messages: Vec<Message>, thinking: bool, tools: Vec<JsonValue> = [], effort: string = "high"): ChatTokens

The tokens of messages in the model's chat template, up to where the assistant's turn starts (ChatTokens).

Effects: fail.

`Model.engine`

gizlang

function Model.engine(self: Model, slots: i64, context: i64, reserve: i64 = 134_217_728, drafts: i64 = -1, rows: i64 = 0, kv: string = ""): Result<Engine, string>

An engine for slots sequences of up to context positions, reading up to rows rows a step (0: 2048 on a GPU, as vLLM's batched tokens, else 256; at least 1 + drafts rows a slot, 2 without drafts). On a GPU the KV pages take the device memory left after the work buffers and recurrent states, at most what the slots can hold, keeping reserve bytes free. Keys and values take the format kv names (kvFormat; F16 where the heads take no codes, see kvWidth), by default the first of kvFormats in which the slots' whole contexts fit above 4 bits, or one whole context at most 4 bits, else the last (on the host 8 bits). An engine reads up to drafts drafts a sequence and step (0: none; 3 is DRAFTS), and with the model's MTP layer Engine.advance drafts them where that gives tokens faster (EXPLORE), on a GPU or the host (where 1 draft is the fastest: its 4-bit layers take two rows in the time of one, and each further draft costs a step of the MTP layer).

Effects: fail, read, write, console, io.

Constants

`CALL`

gizlang

const CALL = "<tool_call>"

Where a model that writes tool calls (Model.calls) starts them.