GizStack
View .mdOpen in ChatGPTOpen in Claude

Language models

gizai/llm runs language models on NVIDIA and AMD GPUs, Apple GPUs, the CPU and an AMD NPU. One engine serves the command line, the OpenAI-compatible server and your programs. This guide is the one reference for it: every option, what it changes, what it costs in quality, and how to measure it.

Your taskSection
Chat with a model on this computerChat
Give an OpenAI client a local modelServe the OpenAI API
Find a model or add oneChoose a model
Make a model folder in the weight format you wantConvert a model
Trade speed, memory and qualityEngine options
Know what the fast settings costQuality
Measure speed and quality on your hardwareMeasure
Call a model from a programUse a model in a program
Change the engineHow it works
Train a model from nothing, on one or many GPUsTraining language models

Before you start

Install Gizlang and run giz doctor. A model downloads and converts once, on its first use: check its license, disk space, memory and GPU memory first. A gated repository needs its license accepted on Hugging Face and HF_TOKEN; HF_ENDPOINT names a mirror.

DeviceWhat runs
NVIDIA GPU (CUDA), sm_80 and laterEvery model and option. INT8 and BF16 tensor-core products; attention by TMA on sm_90 and later.
NVIDIA sm_89 and laterAlso FP8 products (FP8 weights, 8-bit attention values).
NVIDIA sm_120 and sm_121 (RTX 50, GB10)Also NVFP4 products: NV4 weights with their inputs rounded to NVFP4 (W4A4).
AMD GPU (HIP)Matrix-core products on gfx11 and gfx120, portable kernels elsewhere.
Apple GPU (Metal)Matrix products; measured on an M5 Pro (bench/ai/llm).
CPUEvery model but Qwen3.8 Flash, slower. Keys and values in 8 bits, 1 draft a step by default.
AMD NPU (XDNA 2)Gemma 4 E2B and Qwen3.5-9B (including Ornith 1.5 9B), from model.q4nx and xclbins/: one chat, at most 4096 positions.

GIZLANG_DEVICE chooses the device (settings); --cpu runs a command's chat models on the CPU for model.safetensors folders. A model.q4nx folder runs on the NPU.

gizai/llm/convert's convertQwenNpu(from, to) converts a bf16 Qwen3.5-9B Hugging Face folder for the NPU: Q4K projections, q8 head and alpha/beta projections, bf16 embeddings. Supply the unmodified FastFlowLM 1.0.9 Qwen3.5-9B layer.xclbin and lm_head.xclbin in to/xclbins/, with their LICENSE_BINARY.txt. Its host library is not needed. The public FastFlowLM model's main branch has older Q41 blocks; its flm_q4k_high_precision revision has the format these kernels take. Qwen prompts currently run a token at a time; MTP and vision are not implemented on this path. A read appends to the recurrent state or restarts at zero; a partial rewind needs a replay.

Chat

sh
giz ai chat qwen3.5-4b "Why is the sky blue? Two sentences."
text
The sky appears blue because sunlight is scattered by the Earth's atmosphere, and shorter blue wavelengths are scattered
more effectively than longer red wavelengths. This phenomenon, known as Rayleigh scattering, causes the blue light to
reach our eyes from all directions in the sky.
  • The answer goes to standard output as it comes. The first use reports the download and conversion on standard error.
  • Without a prompt, chat reads your messages line by line and keeps the chat. End it with Ctrl-D.
  • Without a model, giz ai chat chats with this computer's default chat model (choose a model).
  • --think lets a thinking model think first, at effort high; --think=low, medium or xhigh give it 2048, 8192 or 32768 tokens instead of 16384. giz env -w AI_EFFORT=LEVEL makes a level the default. The thinking goes to standard error, so the answer stays clean.
  • --max N limits an answer to N tokens. --context N sets the positions of the chat (32768 by default).
  • --temperature T, --top-p P, --top-k K, --min-p P and --seed N replace the model's own sampling (its generation_config.json; sampling). Temperature 0 takes the most likely token each step.
  • Every engine option applies, for example --kv 8 or --draft mtp.
  • chat --cloud MODEL chats with any model of GizStack's AI Gateway instead, billed to the account that giz auth login signed in.

Serve the OpenAI API

sh
giz ai serve qwen3.8-27b --port 8000
text
serving qwen3.8-27b at http://127.0.0.1:8000/v1

Any OpenAI client can use it. Give it the address and the model's name:

sh
curl -s http://127.0.0.1:8000/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model": "qwen3.8-27b", "messages": [{"role": "user", "content": "Hello"}], "stream": true}'
  • The server answers POST /v1/chat/completions (whole or streamed, with tools and reasoning_effort), POST /v1/completions (a text prompt), POST /v1/images/generations and /v1/images/edits (image models, compute guide), POST /v1/embeddings, POST /v1/rerank, GET /v1/models and GET /metrics (Prometheus text: requests running and waiting, KV pages, steps, tokens given out and read, drafts kept, and the GPU memory its engines hold and the card uses).
  • A thinking model thinks at most 2048, 8192, 16384 or 32768 tokens at reasoning_effort low, medium, high or xhigh, and at most half of max_tokens. At that budget the server closes the thinking, and the model goes on.
  • A request takes every sampling field of OpenAI and vLLM, n (up to 16 choices) and stop (strings). The sampling flags (--temperature, --top-p, --top-k, --min-p) set the defaults that a request's own fields replace.
  • response_format {"type": "json_object"} or {"type": "json_schema", "json_schema": {"schema": ...}} lets the model write only JSON of that schema (after its thinking): gizai/llm/grammar masks each token that cannot continue it. A schema's type, properties, required, additionalProperties (false when not given), items, minItems, maxItems, minLength, maxLength, enum, const, anyOf, oneOf and $ref hold; minimum, maximum, exclusiveMinimum and exclusiveMaximum on a plain integer or number (up to 10^18; a bounded number has no exponent, and a fraction only where every fraction of its whole part is inside); and pattern on a plain string: ^ and $, characters, ., classes of ASCII characters with ranges and \d \w \s, and the quantifiers * + ? {n,m} (no groups, alternatives or back references; no \u escape in the string). A oneOf needs choices that no value matches twice (other kinds, other const/enum values, or a required property of other values). A schema with a keyword the grammar does not enforce (format, a pattern outside that subset, ...) is refused with status 400.
  • A message's content parts image_url and video_url (a data URL each: data:...;base64,...) show a picture or a video to a model with a vision tower (Qwen3.5, Qwen3-VL: visual.safetensors in its folder). A picture keeps its aspect in sides of multiples of 32, from 65,536 to 1,048,576 pixels (up to 1024 prompt tokens). A video gives 2 frames a second, at most 32 over the whole video, each at most 200,704 pixels; ffmpeg on the PATH decodes it. Another model refuses such a request. When the GPU has no room for a picture's pass now, the answer is 503 with Retry-After, also for a stream (its status waits for the first piece), so a client can go to another server.
  • /v1/embeddings gives the embeddings of an embed model: input is a text or a list of texts, and encoding_format is float (the default) or base64 (the f32 values, little-endian).
  • /v1/rerank (and /v2/rerank, the Jina and Cohere API) ranks documents for a query with a rank model: results come by relevance_score, the highest first. top_n keeps that many, and return_documents: false leaves out the texts.
  • Name more than one model to serve them together, for example giz ai serve qwen3.5-4b flux3-klein e5-small. The first chat model is the default, and its keys and values take the GPU memory the others leave. One image or one encoding request runs between two steps of the chats.
  • --slots N sets the chats that run together (4 by default) and --context N the positions of each. Without --context, a chat may take every position the GPU holds in 4 bits, up to the model's context: the chats share them, and one that finds no room waits at most 10 s for the chat that ran longest to step aside. A chat's next request reads only its new messages: the engine keeps the rest. --host 0.0.0.0 listens on every address. The server starts after it has compiled and tuned every step shape (Engine.warm), so no request waits for that.
  • ALIAS=NAME serves model NAME under the name ALIAS, as /v1/models lists it and requests give it: giz ai serve coder=/models/q38 --port 8000. Without an alias, a model folder is served by its folder's name.
  • --lora NAME=PATH[:STRENGTH][,PATH...] serves PEFT LoRA adapters (folders or .safetensors files) over the model's own weights, without converting it again; give it once for each. A request whose model is NAME runs with that LoRA, and /v1/models lists it. Requests of many LoRAs and of none run in the same step: each row reads its own adapter's A and B beside the shared quantized product. A chat's keys and values are reused only under the LoRA that made them. LoRAs run on a GPU, on every linear (the head and the MTP layer too). giz ai chat --lora PATH[:STRENGTH] chats with one, or several stacked.
  • --api-key KEY, or GIZ_AI_API_KEY in the environment (which keeps it out of the process list), makes each request to /v1 and /v2 give Authorization: Bearer KEY; other requests get status 401. /metrics stays open.
  • Without a model, giz ai serve serves this computer's default chat model (choose a model).
  • For a coding agent, set its OpenAI base URL to http://127.0.0.1:8000/v1 and its model to the name you served.

Choose a model

sh
giz ai models

The output starts with this computer's GPU and its default chat model, then the catalog: each model's name, kind (text, embed, rank, image, ...), whether it is ready here and whether it is its kind's default, its repository and a short text.

  • The default chat model is the one giz ai chat and giz ai serve run when you name none. On a GPU of 16 GB or less, or with no GPU, it is Ornith 1.5 9B (ornith-1.5-9b); on a larger GPU, Swift 1.5 (swift-1.5), a Qwen3.8 27B tuned for coding agents. Swift 1.5 has its own license (Swift Open License 1.0): read its terms before commercial use. giz env -w GIZ_AI_MODEL=NAME saves another default.
  • The formats follow the GPU. A chat model of 4-bit linears converts them to NVFP4 on an NVIDIA Blackwell GPU (its FP4 matrix products), and to 4-bit codebooks on other GPUs and the CPU. Each format is converted once and kept. NAME:FORMAT names a format yourself: giz ai chat qwen3.8-27b:q8.
  • A drafter comes with the 27B models (Qwen3.8 27B and its fine-tunes): the DFlash 2 block drafter converts beside the model on its first use, and drafts tokens on one GPU (drafts).

A name of the catalog is an entry of a model family, ai/models/NAME/family.giz; a new family is a new folder and giz ai registry (ai/README).

Three other names run too:

  • A Hugging Face repository (owner/name, or its address), run as its files make it: config.json of an architecture below (giz ai chat Qwen/Qwen3-8B "Hello"), converted with 4-bit layers and a 6-bit output layer; a BERT, RoBERTa or XLM-RoBERTa encoder, an embedding model or a reranker (...ForSequenceClassification); or GGUF files, of which the one named by its address (.../blob/main/FILE.gguf) or else the Q4_K_M file. A diffusers pipeline runs only by a family of the catalog. A repository that holds a model already in this library's format (a giz ai convert folder uploaded as it is) downloads and runs as it is, with its drafter/ and tune/ folders: nothing is converted.
  • A model folder, by its absolute path: giz ai serve /models/qwen3-8b-nv4. Make one with giz ai convert to choose its formats.
  • pull NAME downloads and converts a model ahead, so its first chat starts at once.

The architectures that gizai/llm/recipes reads, by model_type of config.json (or, for a renamed model_type without code of its own, by the transformers class its architectures names, as Qwen3ForCausalLM):

Familymodel_type
Llama, InternLM 3, Seed-OSSllama, internlm3, seed_oss
Mistralmistral
Qwen 2, 2.5 and their vision modelsqwen2, qwen2_vl, qwen2_5_vl
Qwen 3qwen3
Qwen3-VL (vision)qwen3_vl
Qwen 3.5 and Qwen3.8 (delta-rule and attention layers, an MTP layer, vision)qwen3_5
Qwen3.8 Flash Next (experts between hyper-connections; one GPU)qwen4_exp
Gemma, Gemma 3, Gemma 4gemma, gemma3, gemma4, gemma4_unified
Phi-3, Granite, GLM-4, SmolLM3, ERNIE 4.5, EXAONE 4phi3, granite, glm4, smollm3, ernie4_5, exaone4
LFM2 and LFM2.5 (short convolution and attention layers)lfm2
BERT, RoBERTa, XLM-RoBERTa encoders (embed, rank)gizai/llm/encoders

A supported architecture does not prove every variant or size; the catalog lists the models that are checked. A new architecture is one recipe: how its config.json reads, and the names of its tensors.

Convert a model

A model runs from a folder in this library's format, or from a Hugging Face folder as it is stored: LlmModel.load reads its safetensors (one file or shards) with no conversion step. Its float linears stay BF16 (F16 when the source is F16), and its FP8 and NVFP4 linears keep their codes. Each linear is laid out in memory as convert would write it, just before it goes to the device. A model with experts or n-gram embeddings, and a vision tower, still load converted. A catalog name, a repository and a GGUF file are converted on their first use (giz ai chat model.gguf "Hello", giz ai serve Qwen3-8B-Q4_K_M.gguf) and kept (the conversion cache). giz ai convert makes a folder you place yourself, from a Hugging Face folder or repository, or a GGUF file:

sh
giz ai convert Qwen/Qwen3-8B /models/qwen3-8b                       # 4-bit layers, 6-bit head
giz ai convert /hf/Qwen3-8B /models/qwen3-8b-q8 --weights 8 --head 8
giz ai convert /hf/Qwen3.8-27B-NVFP4 /models/q27-nv4 --weights nv4   # keeps the checkpoint's NVFP4 values
giz ai convert Qwen3-8B-Q4_K_M.gguf /models/qwen3-8b-gguf
giz ai convert /hf/base /models/tuned --lora /hf/my-adapter
giz ai convert /hf/drafter /models/q27-nv4/drafter                  # a block drafter beside its model
giz ai serve /models/q27-nv4
FlagValuesDefault
--weights FORMATThe linear layers' format (below).4
--head FORMATThe output layer's format.6
--sensitive BITSCodebook bits of the linears nearest the ends of the layer stack, one weight in eight (mixed precision).none
--layers NOnly the first N layers, without the head: a text encoder (Engine.hidden).all
--lora DIRA PEFT LoRA adapter merged into the weights before they are quantized: W + alpha / r · B A (alpha / sqrt(r) with rsLoRA). giz ai train --lora trains one (Training).none
--checkpoint nvfp4Write a standard checkpoint, not a model folder: the float model's layer linears as compressed-tensors stores NVFP4 (NAME_packed, NAME_scale, NAME_global_scale: one scale a tensor), the rest as it is. Convert the checkpoint again, without this flag, to run it: its values are kept. It writes no quantization_config into config.json yet.none
FormatWhat a weight holdsProducts on a GPU
2 to 8A code of 2 to 8 bits of a codebook, after a Hadamard rotation of each block of 128 inputs.Decoded in the kernel.
bf16, f16The value.BF16 or F16 tensor cores.
f8FP8 E4M3 with a scale per row and block.FP8 tensor cores on sm_89 and later.
nv4NVFP4: E2M1 with an E4M3 scale per 16 inputs.NVFP4 tensor cores on sm_120 and sm_121, inputs rounded to NVFP4.

One path for every source

Safetensors and GGUF, float, FP8, NVFP4 and GGUF blocks enter by one path (gizai/llm/plan):

StepWhat it is
WeightViewOne source tensor, whatever its container: name, shape, dtype, its own quantization and scales, its place in its file.
The plannerFor each linear, the QuantSpec it is stored as, and how: its codes laid out again with every value kept (relayout), or its values rounded to the format (requantize). A folder already in this library's format is used as it is.
QuantSpecOne record of how a linear is stored: format, bits, the inputs a scale covers, the scales' kind, the codebook's size, the rotation, the order of the codes, the version. The file names each linear's spec in its metadata.
The productsThe same linears and calls on the CPU, NVIDIA, AMD and Metal. A device's matrix cores choose only the kernel.
SourceStored asValues
Float weights (F32, F16, BF16)The asked format, after the rotation.Rounded once.
NVFP4 (NVIDIA Model Optimizer, compressed-tensors)nv4Kept.
FP8 with a scale a row or a tensorf8Kept.
FP8 with 128 × 128 block scalesf8Rounded to F8 again.
GGUF Q4_K, Q6_K (inputs a multiple of 256)k4, k6: the blocks in the lane order.Kept.
GGUF Q2_K to Q8_0, Q8_K, IQ1_S to IQ4_XSDecoded, then as float weights.Rounded once more.
  • A source of quantized values (NVFP4, FP8, K blocks) gives a model of unrotated inputs: rotating the inputs of a weight needs its values rounded again. Its other linears are stored in f8. No linear is then turned back where it runs. With --lora, a GGUF file keeps the rotation (the merged linears take a codebook), and its untouched K blocks read their inputs turned back.
  • A loader does not guess a format. It reads each linear's spec, refuses a spec of another version, and checks the dtype, shape and size of every tensor against the spec before anything runs. A file of the earlier format (no specs in its metadata) still loads.
  • A GGUF file of Llama, Qwen2, Qwen3, Qwen3.5 or Gemma 4 is read, with its tokenizer and chat template. Other architectures are refused by name. A Qwen3.5 file holds no MTP layer, so it runs without MTP drafts.
  • Measured on the CPU with Qwen3.5 0.8B, 1024 tokens of this guide, keys and values in 16 bits: the Q4_K_M file's conversion (6 s) has -log p 2.981, the original's conversion at --weights 4 3.159. Against the original at 8 bits the file's next-token KL is 0.00 to 0.12 a position on three short texts.

Mixed precision

--sensitive 6 with --weights 4 gives 6 bits to the linears nearest the ends of the layer stack, layer by layer from both ends inward, until they hold one weight in eight: 4.25 bits a weight. The first layers' errors pass through every later layer, and the last layers' go straight to the logits. It is a rule of position, not a measurement: after the rotation, every linear's codebook error is the same within 5%, so the weights alone rank nothing. Its gain in -log p is not measured yet.

A codebook is fitted on 256 rows of its linear, one from each of 256 equal runs of rows, each giving every fourth of its blocks of 128 inputs (a linear of fewer rows gives more of each row's blocks). Against 64 whole rows at even steps (the earlier sample), on 16 linears of Qwen3.5 0.8B the mean error is the same and the worst linear's error over a fit on all its values falls from 7.2% to 2.0% at 4 bits (1.2% to 0.8% at 3 bits).

The conversion cache

What names a conversionWhy
The source's revision (a repository's pinned commit; a GGUF file's path, size and time of change)A new source converts again.
The formats asked (--weights, --head, --sensitive) and the layers keptEach setting has its own folder.
The conversion's algorithm and the layout's versionA new version of this library converts again.
  • The folder holds that key in conversion.json, written last, with the size of model.safetensors.
  • A folder is used only when its key is the one asked and the file has that size. A folder of an earlier version, of another key, or cut short is converted again in its place.
  • A conversion is written to a partial folder that takes its place by a rename. One lock a folder: a second process waits for the first.
  • A vision tower (model.visual.*) goes to visual.safetensors and stays in host memory; a picture moves it over the bus.
  • A model converted from a catalog name or a repository is kept in the models folder; a folder you convert is yours to place.

Measure the KV costs of a model

The engine chooses a format for each attention layer's keys and values by its cost (KV formats). giz ai kvcosts measures that cost once and keeps it in the folder:

sh
giz ai kvcosts /models/q27-nv4 sample.txt --tokens 1024

It prints and writes FOLDER/kv-costs: for each attention layer, the mean KL of the next token when that layer alone keeps its keys and values in each format (8, 8/4, 4, 4/2) and every other layer in 16 bits, on an exact engine, as AutoQuant KV scores them. Without the file, every layer takes the costs measured on Qwen3.8-27B. Use a text like your workload.

Engine options

One record, EngineOptions, holds every option of an engine. giz ai chat, giz ai serve, the benchmark bench/ai/llm/run.giz and your program (Model.engine, Ai.ready) read the same flags. A flag left out takes the best for the model and the device, so the defaults are the tuned setting; set a flag only to trade one thing for another.

FlagEngineOptionsValuesDefaultWhat it changes
--slots Nslots1 or more1 (serve: 4)Sequences that run together. More slots: more tokens a second in total, less KV room each.
--context Ncontextpositionsthe trained context, on a GPU at most what its memory holds in 4 bits (chat: 32768)Positions a sequence. Past the trained context, the rotary table stretches (YaRN).
--kv FORMATkv16, 8, 4, 2, nv4, nv4/VALUES, KEYS/VALUES, or one a layereach layer's of least cost that fitsBytes a position, so contexts that fit, and a small error (KV).
--kv-codes KINDSkvCodesall, nv4, uniformallThe codes the default choice takes: nv4 drops 4-bit keys at even steps, uniform drops NV4 keys (KV).
--fuse KINDSfusenone, or conv, nv comma-separatednoneKernel fusions that were not faster on the RTX 5090: conv convolves inside the delta rule's chunk kernel, nv writes gate_up's activations as the down GEMM's NVFP4 input.
--drafts Ndrafts0 or more32 on a GPU, 1 on the CPUThe most drafts a sequence and step. 0: no drafts. The engine measures and uses the fastest count up to it.
--draft SOURCEdraftblock, mtp, lookupthe first the model hasWhere drafts come from (drafts).
--lookup Nlookup1 or more2The tokens a lookup draft matches before it. More: fewer drafts, kept more often (vLLM's prompt_lookup_min).
--recall Nrecall0 or more8The tokens a recalled draft matches before it (drafts). 0: no recall.
--rows Nrowsrows2048 on a GPU (512 on one below 12 GiB), 256 on the CPUThe most rows a step reads: the prompt chunk. More rows: a faster prefill on a large GPU, more work memory.
--prefix on|offprefixon, offonWhether a prompt reuses the keys and values of a start that an earlier prompt had.
--reserve BYTESreservebytes134217728 (128 MiB)GPU memory the engine leaves free for other programs.
--exactexactflagoffThe linears take their inputs as computed: a slow reference (precision).

A flag given more than once sweeps in the benchmark: --kv 8 --kv 4 --drafts 0 --drafts 8 runs all four combinations and marks the best (measure).

Keys and values (KV)

Each attention layer keeps the keys and values of every position in pages of 16 positions. A format is the bits of a key and of a value:

FormatSize against 16 bitsUse
161F16: the reference.
8about 1/2The output of 16 bits (KL 0.00003, quality).
4about 1/4Twice the context of 8.
4/2about 1/5Keys in 4 bits, values in 2: the most context.
nv4/4, nv4/8about 1/4, 3/8Keys in NVFP4: FP4 products with the queries on RTX 50 GPUs (sm_120).
nv4about 1/4Keys and values in NVFP4: FP4 products with the queries and with the weights (sm_120, heads of 256 dims).
8,8/4,4,...per layerOne format for each pool layer, in order.
  • A code holds 2^bits even steps over each head's range, after a rotation of each block of the head; the range is the one that gives the head back with the least squared error. A head of a size other than 64, 128, 256 or 512 stays in F16.
  • nv4 keys are NVFP4: an E2M1 code a value of the rotated head, an E4M3 scale a block of 16 dims and an F32 scale a head. On sm_120 the matrix cores multiply the codes as they are with NVFP4 queries (FP4 products, twice the FP8 rate). Elsewhere attention decodes them, so the default choice takes them only on sm_120.
  • nv4 values (with nv4 keys, heads of 256 dims; other heads take 4 bits) are E2M1 codes of the rotated head and an F32 scale a head. On sm_120 the weights of a row become NVFP4 too, and the matrix cores multiply them with the codes as they are: no warp decodes keys or values.
  • Values lose bits first: a key's error moves every score of its position, a value's only its share of the sum.
  • By default the engine gives each layer the format of least summed cost (the folder's kv-costs) in which the most whole contexts of the slots fit at 4 bits or more; only where none fits it allows 2 bits. A per-layer list needs one format for each pool layer: the attention layers, and the MTP layer when it drafts.
  • With codes, attention reads 8-bit queries against the key codes in integer products (NV4 keys: FP4 products) and the values in FP8; with 16 it runs in F16.
  • The CPU keeps 8 bits by default.

Drafts

A step can read drafts after each sequence's token: guesses of the next tokens. The model checks them all in the same step and keeps those it agrees with, so a step gives several tokens. The tokens follow the same distribution as without drafts. Greedy output can still differ in a rare place, because a step of more rows sums in another order.

--draftSourceNeeds
blockA block drafter (DFlash, DFlash 2): a small model that drafts a whole block in one pass from the target's hidden states of a few layers over the last 2048 positions. Its selector picks each draft from the 16 most likely tokens.drafter/ in the model's folder (convert), the model on one GPU.
mtpThe model's multi-token prediction layer: one draft a pass of it. A draft reads an eighth of the output layer, the rows of frequent tokens and of the tokens read last.A model with an MTP layer (Qwen 3.5, Qwen 3.8).
lookupThe tokens that came after the sequence's last 2 tokens (--lookup) where they came before in its own text.Nothing.
  • By default the engine takes block where the folder has a drafter, else mtp where the model has the layer, else lookup. The model loads only the weights of the source you name: --draft lookup or --drafts 0 leaves the MTP layer and the drafter out of GPU memory, so more KV fits.
  • With block and mtp, recall adds drafts: where a sequence's last 8 tokens or more (--recall; 0 turns recall off) came before in its text (a code edit, a quote) and the source's first draft agrees, or the match is 32 tokens or more, the tokens after the match take the whole depth.
  • The engine drafts the count that gives the most tokens a second: the tokens a step it expects over the time a step of that count took, for each load (sequences generating) and context length. It judges every draft a source made against the tokens that follow, also those the step did not read, so it knows what a longer count would keep. A count it has not measured runs once. --drafts N caps the count.
  • On Qwen3.8-27B (RTX 5090) a step of 7 block drafts gives about 6.4 tokens, against about 4.6 for the MTP layer. A 27B model on one RTX 5090 has the whole setup and its numbers.
  • A request whose logits are shaped (sampling) makes no drafts.

Prefix reuse and shared heads

  • A chat's next request reads only its new messages: Model.chatTokens marks where each message ends, and the slot keeps its keys, values and recurrent states there. When another chat takes the slot, the old start stays in the KV pool until the pool needs the room, then in host memory, and the chat's next turn takes it back. Past the host memory free, the oldest starts of a GPU model go to files in GIZLANG_CACHE/kv (one folder for each model folder and the KV page layout of its stages, a model in stages filing each stage's pages beside the first's; the pages of all of them share GIZLANG_KV_GB, 16 GiB by default, the oldest out first), and a later start of the server takes them too. A filed start comes back from its page files straight to the card, the pages read together (Gpu.readFiles).
  • A Flash (MoE from the disk) model keeps a chat's start too, with its PLE states; it is taken back whole, as on a model with recurrent layers, and slots do not share heads in the pool.
  • A system prompt with tools of 256 tokens or more is a shared head: every slot reads one copy of its keys and values.
  • --prefix off turns all of this off: each prompt is computed whole, as a server without a prefix cache.

Precision

  • On sm_120 and sm_121, NV4 linears round their inputs to NVFP4 by row and run NVFP4 products (W4A4), as standard NVFP4 serving does. FP8 linears run FP8 products on sm_89 and later. Codebook linears decode their codes in the kernel.
  • --exact gives the linears their inputs as computed (F32) and runs NV4 and FP8 as matrix-vector products: the reference of quality, many times slower. There is no faster high-precision mode yet.

Long contexts

  • --context past the trained context stretches the rotary table by YaRN for that engine. On 2K tokens, a factor of 2 cost KL 0.0025 and a factor of 4 cost 0.0068 (Qwen3.8-27B).
  • A step reads at most --rows prompt rows. While other sequences generate, it reads at most the prompt rows that add as much time as their step alone: a long prompt slows them to half their pace at most.

Many GPUs

A model that fits one GPU serves more requests on several: giz ai serve qwen3.8-27b --gpus 0,1,2,3 --port 8000 runs a server of the models on each GPU (its CUDA_VISIBLE_DEVICES, at the ports after --port on 127.0.0.1) and answers at --port once each listens. A request goes to the replica with the fewest requests in flight; a conversation's next request goes back to the replica that had it, whose engine keeps its keys and values, while that one holds at most 4 requests more than the least busy. Answers stream through as they come, /metrics gives every replica's samples with a replica label, and a replica that ends ends the server.

A model whose weights leave less than 1 GiB of its GPU free loads across the other NVIDIA GPUs that CUDA_VISIBLE_DEVICES shows: each GPU holds a stage of consecutive layers, in proportion to its free memory, and a step runs the stages in turn. The last stage holds the head and the MTP layer and drafts; a block drafter needs the model on one GPU. Each stage keeps its own KV pages. On two RTX 4080s, a decode hop of 4 rows took 10–40 µs and a 2048-row prefill hop 3–4 ms. loadAcross(folder, gpus, counts) chooses the GPUs and the layer counts.

Qwen3.8 Flash

Qwen3.8 Flash (experts and hyper-connection streams) runs on one GPU. Its experts are read from the model file as steps need them into an expert cache, which takes the GPU memory left after the work buffers, --reserve and 2 GiB.

Sampling

Each request picks its tokens by its own sampling. A model's generation_config.json gives the defaults; a model without that file takes its family's recommended sampling (Qwen3.5: temperature 1, top-k 20, top-p 0.95), or the most likely token where the family has none. The chat and serve flags replace them, and a request's fields replace those. One table names each control in all three places:

ControlFlag (chat, serve)API fieldSampling fieldShapes the logits
Temperature (0: the most likely token)--temperature Ttemperaturetemperatureno
Keep the K most likely (0: all)--top-k Ktop_ktopKno
Keep the most likely whose probabilities reach P--top-p Ptop_ptopPno
Keep the tokens at least P times as likely as the most likely--min-p Pmin_pminPno
Random seed--seed N (chat)seedseedno
Less likely each time the answer has the tokenfrequency_penaltyfrequencyyes
Less likely once the answer has the tokenpresence_penaltypresenceyes
Divide the logit of a token the prompt or the answer hasrepetition_penaltyrepetitionyes
Add to a token's logit (-100 bans it)logit_bias ({"TOKEN": VALUE})biasyes
No stop token before N tokensmin_tokensminTokensyes
Only JSON, or JSON of a schemaresponse_formatgrammaryes
Log probabilities and the most likely tokenslogprobs, top_logprobslogprobsyes
More tokens that end the answerstop_token_idsstopTokensno
The model's own stop tokens do not end the answer (a fixed length)ignore_eosignoreEosno
A program's own rule over the logitsshapeyes
  • top_k, min_p, repetition_penalty, min_tokens, stop_token_ids and ignore_eos are vLLM's fields; the rest are OpenAI's. A value out of range (min_p above 1, a logit_bias token the model lacks) gets status 400.
  • Temperature, top-k, top-p and min-p choose from the model's distribution, so a request with only these still drafts. A request that shapes the logits makes no drafts: each step reads its token alone.

Quality

What the fast settings cost, against the same NVFP4 weights run exactly (--exact, 16-bit keys and values). Qwen3.8-27B NVFP4, an RTX 5090, an English text, 2026-11-08 (run.giz quality; KL of the next-token distribution, top 1 agreement, mean -log p of the text's next tokens):

SettingTokensKLTop 1-log p (exact → setting)
Keys and values in 8 bits, exact linears4K0.0000399%1.974 → 1.974
The tuned KV plan per layer, exact linears4K0.003097%1.974 → 1.978
NVFP4 inputs to the linears (W4A4), 16-bit KV4K0.031992%1.974 → 1.989
NVFP4 inputs to the linears (W4A4), 16-bit KV16K0.033991%1.569 → 1.583
W4A4, the KV plan (the default --kv), 2048-row steps4K0.034088%1.974 → 2.001
W4A4, the KV plan, 2048-row steps16K0.034091%1.569 → 1.547
W4A4, the KV plan, 16-row steps (as decode)4K0.027292%1.979 → 1.965
W4A4, --kv nv4, 2048-row steps (the fastest on sm_120)4K0.039390%1.974 → 1.962
W4A4, --kv nv4, 2048-row steps16K0.040691%1.569 → 1.577
  • Nearly all of the error is the NVFP4 rounding of the linears' inputs, the same as any W4A4 NVFP4 server. The KV plan adds KL 0.002 at 4K and nothing measurable at 16K; nv4 keys and values add about 0.007. Windows of 127 rows move by about 0.003 between runs.
  • A fact at 10%, 50% and 90% of 32K tokens and at 50% of 100K was answered right with 16-bit keys and values, with the plan and with nv4 (12 of 12).

Measure

bench/ai/llm/run.giz measures a model folder. Build it once and run it on the GPU host (bench/ai/llm has the published results):

sh
bin/gizlang build bench/ai/llm/run.giz -o llmrun
./llmrun speed /models/q27-nv4 1 8                    # prefill of 1K tokens, decode of 1 and 8 sequences
./llmrun speed /models/q27-nv4 1 8 --drafts 0 --drafts 32   # a sweep: each, then a table
./llmrun chat /models/q27-nv4 "Explain rotary embeddings." 256 --profile
./llmrun prefill /models/q27-nv4 2048 4 32768        # a 2048-row step after 32K positions, by kernel
./llmrun quality /models/q27-nv4 text.txt 4096 --kv 8 --exact
./llmrun needle /models/q27-nv4 text.txt 32768 50    # a fact at 50% of 32K tokens, then the answer
./llmrun load http://127.0.0.1:8000/v1 8 64          # 8 chats at once against a server
CommandWhat it prints
check FOLDER [TOKENS]The GPU's logits against the CPU's after a prompt.
chat FOLDER TEXT|@FILE [TOKENS [TEMPERATURE]]A reply, its speed and the drafts kept; --profile: its kernels a step.
speed FOLDER [BATCH...]Prefill tokens/s of a 1024-token prompt, then decode tokens/s per batch size.
profile FOLDER [BATCH [STEPS [AFTER]]]The GPU time of decode steps by kernel.
prefill FOLDER ROWS [STEPS [AFTER]]The GPU time of a prompt step of ROWS rows after AFTER positions, by kernel.
quality FOLDER FILE [TOKENS]The engine of the flags against the exact one: KL, top 1, -log p, by window and in all. --rows reads that many rows a step.
needle FOLDER FILE TOKENS [DEPTH]KV positions, prefill and decode speed at that length, and the answer about a fact at DEPTH %.
load URL CLIENTS REQUESTS [TOKENS [PROMPT]]Tokens a second, the median first-token time and the share of later prompts read from the server's cache.
  • Every command takes the engine flags. speed turns prefix reuse off, so a repeated prompt is computed whole.
  • Every run keeps the kernel tunings it measured in FOLDER/tune; later runs and the server take them.
  • GIZLANG_LLM_PROBE set prints each step's rows, drafts, tokens and time, and each stage's hop (settings).

Use a model in a program

gizai/ai gives a program the models by name. Ai.models opens the models folder, Ai.text loads a chat model, Ai.ready makes its engine warm, and Ai.reply gives the answer piece by piece:

typescript
import {Message} from "gizlang/openai";
import * as Ai from "gizai/ai";
import {EngineOptions} from "gizai/llm/engine";

function main(): void {
    const hub = Ai.models()!;
    const options: EngineOptions = { context: 8192, kv: "8" };
    const m = Ai.text(hub, "qwen3.5-4b", true, options)!;
    let e = Ai.ready(m, options)!;
    const asked: Vec<Message> = [{ role: "user", content: "Name three rivers." }];
    const answer = Ai.reply(e, asked, Ai.ask(), (p: Ai.Piece): bool => true)!;
    console.log(answer.text);
}
  • Pass the same EngineOptions to Ai.text (it loads only the named draft source's weights) and to Ai.ready.
  • Ai.Ask holds what a reply may do: its most tokens, stop strings, reasoning effort, tools (OpenAI's tool objects) and sampling. A Piece holds the text, the reasoning, tool calls, log probabilities and why it ended.
  • Ai.encoder(hub, name) loads an embed or rank model; gizai/serve's Serve.serve(listener, models, images, encoders, key) is the server, with the API key key ("" for none).
  • gizdesk/studio (GizDesk's, AGPL-3.0) connects local models and model APIs to agents (agent guide).

gizai/llm is the layer below: convert, load, and run an engine of many sequences:

typescript
import {convert} from "gizai/llm/convert";
import {load} from "gizai/llm/model";
import {greedy} from "gizai/llm/sample";

function main(): void {
    convert("Qwen3-8B", "qwen3-8b-q4", { layers: 4, head: 6 })!;
    const m = load("qwen3-8b-q4")!;
    const prompt = m.tokenizer.tokens(m.chatPrompt([{ role: "user", content: "Name three rivers." }], false));
    let e = m.engine({ context: prompt.length + 200 })!;
    const out = e.generate([{ prompt: prompt, limit: 200, sampling: greedy() }])!;
    console.log(m.tokenizer.decode(out[0], true));
}
  • load(folder, onGpu, layers, options) loads on the best device; with layers, only the first layers, and Engine.hidden gives each token's hidden state after them (a text encoder of an image model).
  • One Engine serves many sequences: pass more requests to generate, or run them in turn with Engine.add, advance and cancel (a request joins while others run, and its tokens come step by step). Engine.add(request, mark, head) keeps a chat's start and a shared head (prefix reuse).
  • A request's Sampling picks its tokens: greedy(), or temperature with topK, topP, minP and seed. Its optional fields are the rest of the sampling table. A grammar of gizai/llm/grammar is anyObject() or fromSchema(schema), from the token after after on; with logprobs each Progress has the tokens' log probabilities and that many of the most likely.
  • shape is your own rule: a function that changes the logits of the whole vocabulary, given the prompt and the answer so far. It runs after the other controls, before the token is picked:
typescript
import {Model} from "gizai/llm/state";
import {Sampling, greedy} from "gizai/llm/sample";

/** Sampling that gives only digits and the stop tokens: an answer that is a number. */
export function digitsOnly(m: Model): Sampling {
    let allowed: Vec<bool> = new Array(m.arch.vocab).fill(false);
    const pieces = m.tokenizer.tokenBytes();
    for (const t of range(0, pieces.length)) {
        let digits = pieces[t].length > 0;
        for (const b of pieces[t]) { if (b < 48u || b > 57u) { digits = false; } }
        allowed[t] = digits;
    }
    for (const t of m.stops) { allowed[t] = true; }
    let s = greedy();
    s.shape = (logits: mut Vec<f32>, tokens: Vec<i64>): void => {
        for (const t of range(0, logits.length)) { if (!allowed[t]) { logits[t] = -1.0e30; } }
    };
    return s;
}
  • Call Engine.warm (or use Ai.ready) before requests: the first run of each step shape compiles, tunes and records its kernels, and Model.saveTunings keeps the tunings for the next start.
  • engineFlags(args) reads EngineOptions from command-line flags, and ENGINE_FLAGS lists them.

Extend

Each kind of change has one place:

To addChangeThen
A sampling rule of your ownSampling.shape in your program (above)Nothing else: it runs on every engine.
A model to the catalog, by nameAn entry of a family in ai/models/NAME/family.giz (gizai README)giz ai registry; models, pull, chat and serve find it.
An architecture (a new model_type)A recipe in gizai/llm/recipes: it reads config.json into the features of an Arch (gizai/llm/shape)A tiny model of it in tests/ai/llm/tiny.giz, checked against the f32 reference.
A weight formatgizai/quant (parseFormat) and its products in gizai/llm/gemm--weights NAME of convert takes it.
A KV formatkvFormats in gizai/llm/kvformat and its attention in gizai/llm/kv--kv and kvcosts take it.
An engine optionA field of EngineOptions and its flag in ENGINE_FLAGS (gizai/llm/engine)chat, serve, the benchmark and programs read it; list it in giz help ai (tools/cli/commands.giz).
A draft sourceDRAFT_SOURCES in gizai/llm/engine, its weights in LlmModel.load and its drafts in gizai/llm/generate--draft NAME; the quality row of the benchmark shows that it keeps the distribution.

A model whose architecture a recipe reads needs no code: giz ai convert and giz ai chat /path/to/folder run it.

Where the files are

The models folder is $GIZLANG_CACHE/huggingface, else ~/.cache/gizlang/huggingface.

PathWhat it holds
OWNER/NAME/The downloaded files: JSON, safetensors, tokenizer and template files. Never code.
.pins/OWNER/NAME.pinThe commit of the downloads and the SHA-256 of each file. A later download of the same model takes the same commit.
.gizlang-llm/OWNER/NAME/ID/qL-H/The model converted with L bits for its layers and H for its output layer (qL-H-sS with sensitive bits). ID names the pinned commit. conversion.json holds the key that says whether the folder is complete and current (the conversion cache).
.gizlang-llm/files/ID/NAME.gguf.qL-H/A GGUF file converted on its first use. ID names its path, size and time of change.

A model folder holds:

FileWhat it holds
model.safetensorsThe linears in their formats, the norms and small tensors in F32, the embedding as its source held it. Its metadata names the format (gizlang.llm.2) and each linear's QuantSpec.
config.json, tokenizer.json, generation_config.json, chat templateThe source's files: architecture, tokens, sampling, chat format.
visual.safetensorsThe vision tower, where the model has one.
drafter/A converted block drafter.
kv-costsEach attention layer's measured cost in each KV format (giz ai kvcosts).
tune/The kernel tunings measured on this machine.
  • A download is checked against the SHA-256 that Hugging Face states, and a stopped download leaves no file.
  • To remove a model, remove its folders. Its next use downloads it again.

How it works

A step reads rows of many sequences at once: prompt chunks and decode rows with their drafts. Each layer runs as GPU kernels on all the rows; attention reads each sequence's paged keys and values. The engine then samples or checks each sequence's tokens, keeps the drafts it agrees with, and plans the next step.

ModuleOwns
gizai/llm/recipes, arch, shapeEach family's config.json read into an Arch: layers, heads, norms, windows, rotary settings, tensor names.
gizai/llm/planWeightView (a source tensor), the planner (each linear's QuantSpec and whether its values are kept), the kernel family a device takes.
gizai/llm/convert, visionconvertConversion to the folder format as the plan says: rotation, codebooks, mixed precision, LoRA, drafters, vision towers.
gizai/llm/modelLoading a folder on one or more devices by its specs, the conversion cache, the chat template, tunings and KV costs.
gizai/llm/stateThe records: Model, Engine, requests, drafts, slots. It imports no other part.
gizai/llm/engineEngineOptions, Model.engine, a step, the KV choice and kvMeasured.
gizai/llm/generateRequests in progress, the prefix cache, draft counts and recall, warming.
gizai/llm/gpu, gemmA step's buffers and launches on a GPU; the products of each weight format, tuned per shape.
gizai/llm/kv, kvformatPaged attention over coded keys and values; the page layout and the per-layer choice.
gizai/llm/deltaThe gated delta rule (Qwen 3.5 and 3.8 recurrent layers) and its replay of kept drafts.
gizai/llm/blockThe block drafter on a GPU.
gizai/llm/sample, grammarSampling, logit shaping, the check of drafts, JSON-schema grammars.
gizai/llm/encodersBERT-family encoders.
gizai/llm/vision, flash, npuVision towers, Qwen3.8 Flash, the NPU path.
gizai/llm/hostThe CPU path of every operation: the reference of the GPU path.

The products, attention and normalization kernels are written in Gizlang with gizhw/kernel and compiled for each device at the first run of a shape (compute internals). A change to the engine keeps the output distribution: measure it with quality and needle against the previous build on the same base.