# gizml:ai AI models on this computer, by name. A name is one of the catalog's (Qwen 3.5 chat models, FLUX.3 klein images) or a Hugging Face repository ("owner/name") of a family that gizml:llm reads. The first use downloads the model into the models folder (gizlang:hub: $GIZLANG_CACHE/huggingface, else ~/.cache/gizlang/huggingface; $HF_TOKEN for a gated repository, $HF_ENDPOINT for a mirror) and converts it once; later uses need no network. `text` gives a chat model, `ready` its warm engine, `reply` an answer as it comes, `image` a text-to-image model, and `serve` the OpenAI API of both. Each family's recipe stays in its module (gizml:llm, gizml:diffusion); this one finds, fetches and serves. An absolute path names a converted model's folder instead. ```gizlang import * as Ai from "gizml:ai"; ``` ## Types ### `Entry` ```gizlang type Entry = { name: string; repo: string; kind: string; bits: L.Bits; about: string; } ``` A model by name: its Hugging Face repository, its kind ("text": a chat model; "image": text to image; "embed": a sentence encoder; "rank": a cross-encoder), the bits of its linears (an image model's: 8, or 4 for INT4; an encoder's are as stored) and what it is. - `name`: `string` - `repo`: `string` - `kind`: `string` - `bits`: `M23.Bits` - `about`: `string` ### `Encoding` ```gizlang type Encoding = { name: string; encoder: Option; ranker: Option; } ``` A text encoder a server serves (serve): the name requests give it, and its sentence encoder (kind "embed") or its cross-encoder (kind "rank"). - `name`: `string` - `encoder`: `_gizlang_sdk_core.Option` - `ranker`: `_gizlang_sdk_core.Option` ### `Ask` ```gizlang type Ask = { limit: i64; stops: Vec; effort: string; tools: Vec; sampling?: Sampling; } ``` What a reply may do: its most tokens (0: the context's rest), the strings that end it, the reasoning effort ("": none; low, medium or high: a thinking model thinks first), the tools it may call (OpenAI's tool objects) and its sampling (none: the model's). - `limit`: `i64` - `stops`: `Vec` - `effort`: `string` - `tools`: `Vec` - `sampling`: `_gizlang_sdk_core.Option` ### `ImageModel` ```gizlang type ImageModel = { name: string; hub: Hub.Hub; gpu: Gpu; klein: Vec; zimage: Vec; tunings: string; need: ImageNeed; upsampler: Ref; } ``` An image model on a GPU, making images from an ImageRequest (`make`): FLUX.3 klein (`klein`: gizml:diffusion's pipeline of its text encoder, Qwen3-VL-4B's first layers, and its VAE) or Z-Image Turbo (`zimage`); one of the two holds the model. - `name`: `string` - `hub`: `gizlang_hub.Hub` - `gpu`: `M10.Gpu` - `klein`: `Vec` - `zimage`: `Vec` - `tunings`: `string` - `need`: `M18.ImageNeed` - `upsampler`: `Ref` ### `ImageRequest` ```gizlang type ImageRequest = { prompt: string; pictures: Vec; width: i64; height: i64; megapixels: f64; steps: i64; seed: i64; count: i64; cache: f64; refresh: i64; upsample: bool; } ``` What is asked of an image model: the `prompt`; the `pictures` to edit (at most 4; none: text to image); the size, `width` × `height` pixels (multiples of 16, at most 4 megapixels; 0: 1024 × 1024 for text to image, else the first picture's aspect at `megapixels`); the `steps` (0: the model's, 16 for klein and 8 for Z-Image); `seed`: the first of `count` images' noise seeds (seed, seed + 1, ...: the same seed gives ComfyUI's image, D.fluxNoise); `cache`, the step cache's limit (0 to 0.5: more is faster and a little less exact); `refresh`, the steps between the pictures' key and value updates (0: never needed); and `upsample`, the prompt written out in full by BFL's upsampler first. - `prompt`: `string` - `pictures`: `Vec` - `width`: `i64` - `height`: `i64` - `megapixels`: `f64` - `steps`: `i64` - `seed`: `i64` - `count`: `i64` - `cache`: `f64` - `refresh`: `i64` - `upsample`: `bool` ### `ImageOutput` ```gizlang type ImageOutput = { extension: string; quality: i64; } ``` How images are written: the file extension ("webp", "png" or "jpg") and the WebP or JPEG quality. - `extension`: `string` - `quality`: `i64` ### `ImageNeed` ```gizlang type ImageNeed = { fixed: i64; perRow: i64; perToken: i64; } ``` What a request needs beside the model, measured at load (`measure`): for its steps, `fixed` bytes and `perRow` bytes for each row of its joint sequence (the image's tokens, the text's and the pictures'); for its decode, `perToken` bytes for each image token the decoder holds at once (at most D.IMAGE_WHOLE: a larger image decodes in tiles). - `fixed`: `i64` - `perRow`: `i64` - `perToken`: `i64` ### `Made` ```gizlang type Made = { images: Vec; prompt: string; } ``` What `make` made: the images, and the prompt they were made from (the upsampler's when `upsample`). - `images`: `Vec` - `prompt`: `string` ### `VideoModel` ```gizlang type VideoModel = { name: string; gpu: Gpu; ltx: X.Ltx; files: VideoFiles; encoderHost: string; encoderPort: i64; authorization: string; tunings: string; duration: X.DurationHead; } ``` A video model on a GPU: LTX-2.5 22B distilled (gizml:ltx), with its text from giz's shared encoder. - `name`: `string` - `gpu`: `M10.Gpu` - `ltx`: `M24.Ltx` - `files`: `M18.VideoFiles` - `encoderHost`: `string` - `encoderPort`: `i64` - `authorization`: `string` - `tunings`: `string` - `duration`: `M24.DurationHead` ### `VideoFiles` ```gizlang type VideoFiles = { transformer: string; videoVae: string; audioVae: string; durationHead: string; upsampler: string; fidelity: string; control: string; pose: string; ingredients: string; repaint: string; } ``` LTX-2.5's files: the transformer (INT8 ConvRot), the video VAE, the audio VAE, the duration head, the latent upsampler, the fidelity decoder (the diffusion VAE), the union control IC-LoRA, DWPose's folder, and the ingredients and in/outpainting IC-LoRAs. - `transformer`: `string` - `videoVae`: `string` - `audioVae`: `string` - `durationHead`: `string` - `upsampler`: `string` - `fidelity`: `string` - `control`: `string` - `pose`: `string` - `ingredients`: `string` - `repaint`: `string` ### `VideoRequest` ```gizlang type VideoRequest = { prompt: string; width: i64; height: i64; frames: i64; seed: i64; image: Option; end: Option; video: string; source: i64; keys: Vec; at: Vec; auto: bool; quality: bool; audio: string; fidelity: bool; control: string; controlMode: string; subject: Option; repaint: string; mask: string; } ``` What is asked of a video model: the `prompt`; `width` x `height` pixels (multiples of 32, at most 1920 x 1088) and `frames` (8 k + 1, at most 481) at 24 a second; the noise's `seed`; the first frame `image` and the last `end`, or a `video` (its path) that the new frames extend: its last `source` frames (8 k + 1) at 24 a second open the output. - `prompt`: `string` - `width`: `i64` - `height`: `i64` - `frames`: `i64` - `seed`: `i64` - `image`: `_gizlang_sdk_core.Option` - `end`: `_gizlang_sdk_core.Option` - `video`: `string` - `source`: `i64` - `keys`: `Vec` - `at`: `Vec` - `auto`: `bool` - `quality`: `bool` - `audio`: `string` - `fidelity`: `bool` - `control`: `string` - `controlMode`: `string` - `subject`: `_gizlang_sdk_core.Option` - `repaint`: `string` - `mask`: `string` ### `Piece` ```gizlang type Piece = { index: i64; text: string; reasoning: string; logprobs: Vec; calls: Vec; end: string; prompt: i64; cached: i64; tokens: i64; } ``` A piece of an answer (choice `index` of a request): more of its text and reasoning and the log probabilities of its tokens (when asked); the last also the tools it calls, why it ended ("stop", "tool_calls", "length", or "error" with the message in text), and the prompt's tokens, those of them read from cache and the tokens written. - `index`: `i64` - `text`: `string` - `reasoning`: `string` - `logprobs`: `Vec` - `calls`: `Vec` - `end`: `string` - `prompt`: `i64` - `cached`: `i64` - `tokens`: `i64` ### `Served` ```gizlang type Served = { name: string; engine: Engine; } ``` A chat model a server serves (serve): the name requests give it, and its engine (ready). - `name`: `string` - `engine`: `M23.Engine` ## Functions ### `catalog` ```gizlang function catalog(): Vec ``` The models known by name, the smallest first. Each fits one GPU of 16 GB with room for its keys and values. Effects: fail. ### `find` ```gizlang function find(name: string): Entry throws string ``` The catalog's model of `name` (or of its repository), else the Hugging Face repository `name` ("owner/name") as a chat model with linears of 4 bits and a head of 6. A chat model's name may end in ":FORMAT" (Q.parseFormat: "qwen3.5-0.8b:bf16"): its linears and head in that format. Effects: fail. ### `models` ```gizlang function models(sys: System): Hub.Hub throws string ``` The models folder (Hub.folder), downloads on. Effects: fail, io. ### `pull` ```gizlang function pull(hub: Hub.Hub, name: string): string throws string ``` The model of `name` (find) downloaded and converted, as its first use does; returns its local folder. Effects: fail, read, write, console, io. ### `here` ```gizlang function here(hub: Hub.Hub, e: Entry): bool ``` Whether the models folder holds model `e` ready: a use downloads and converts nothing. Effects: fail, io. ### `encoder` ```gizlang function encoder(hub: Hub.Hub, name: string, onGpu: bool = true): Encoding throws string ``` The encoder of `name` (find; kind "embed" or "rank"), from its repository's files (L.encoderOf, L.crossEncoderOf), on the GPU when `onGpu` and there is one. Effects: fail, read, write, console, io. ### `text` ```gizlang function text(hub: Hub.Hub, name: string, onGpu: bool = true): Model throws string ``` The chat model of `name` (find), on the GPU when `onGpu` and there is one. Effects: fail, read, write, console, io. ### `ready` ```gizlang function ready(sys: System, m: Model, slots: i64 = 1, context: i64 = 0, kv: string = "", drafts: i64 = -1): Engine throws string ``` An engine of `m` for `slots` chats of `context` positions each (0: the model's context), keys and values in the format `kv` names (Model.engine; "": the best that fits), `drafts` tokens drafted a step (Model.engine; -1: L.DRAFTS), warm: each step shape compiled and tuned before the first request, the tunings kept in the model's folder for the next start (a file for each GPU model, Gpu.tuningsFile: a model in stages may span kinds). Effects: fail, read, write, console, io. ### `ask` ```gizlang function ask(): Ask ``` An Ask with no limit, no stops, no reasoning, no tools and the model's sampling. ### `reply` ```gizlang function reply(sys: System, e: mut Engine, messages: Vec, a: Ask, each: (piece: Piece) => bool): Piece throws string ``` The answer of `e`'s model to `messages`, each piece given to `each` as it comes (false: stop there). Returns the whole answer as one piece: its text, reasoning, tool calls, why it ended and the token counts. Effects: fail, read, write, console, io. ### `image` ```gizlang function image(hub: Hub.Hub, name: string = "flux3-klein"): ImageModel throws string ``` The image model of `name` (find; "flux3-klein"). It needs a GPU. Effects: fail, read, write, console, io. ### `imageRequest` ```gizlang function imageRequest(prompt: string): ImageRequest ``` A text-to-image request for `prompt` with every other setting at its default. ### `checkImage` ```gizlang function checkImage(r: ImageRequest): bool throws string ``` Whether `r` is a request an image model takes: the one place that says what is too big, too many or out of range. Effects: fail. ### `imageRequestOf` ```gizlang function imageRequestOf(j: JsonValue, pictures: Vec, seed0: i64): ImageRequest throws string ``` The request of a JSON object (or a form's text fields) and the pictures that came with it: `prompt`, `size` ("WIDTHxHEIGHT") or `width` and `height`, `megapixels`, `steps`, `seed` (`seed0` when none), `count`, `n` or `numberResults`, `stepCache`, `refresh` and `upsample`. It is not checked (checkImage). Effects: fail. ### `imageOutputOf` ```gizlang function imageOutputOf(j: JsonValue): ImageOutput throws string ``` The output of a request's `outputFormat` (or OpenAI's `output_format`): webp (the default, a tenth of a PNG's bytes), png or jpeg, at `outputQuality` (or `output_compression`; 90). Effects: fail. ### `serveImages` ```gizlang function serveImages(sys: System, m: ImageModel, host: string, port: i64, state: string, capacity: i64 = 4): i64 throws string ``` Serves image model `m` as giz's inference worker (gizlang:worker, protocol v1) on host:port, its jobs under `state`: the one model `m.name`, operation image.generate, `capacity` jobs waiting. It returns when the worker stops. Effects: fail, read, write, console, io. ### `videoFiles` ```gizlang function videoFiles(sys: System, folder: string): VideoFiles throws string ``` LTX-2.5's files in a ComfyUI models folder. Effects: fail, io. ### `video` ```gizlang function video(sys: System, name: string, files: VideoFiles, encoder: string, authorization: string, tunings: string): VideoModel throws string ``` LTX-2.5 (`name`) of `files` on this machine's GPU, its text from the encoder at `encoder` (HOST:PORT), the tunings of its kernels kept in the folder `tunings` (the next start needs none). Effects: fail, read, write, console, io. ### `videoRequestOf` ```gizlang function videoRequestOf(sys: System, j: JsonValue, seed: i64): VideoRequest throws string ``` The request of a job's input as giz's ltx2-5-distilled sends it: `prompt`, `resolution` ("832x480"), `frames` (97), `seed`, the uploads `image`, `imageEnd` and `video`, and `injectedFrames` at `injectedFramePositions` ("0, 48, L": pixel frames, L the last). Workflows multishot and story are standard ones (their cuts are in the prompt). Its fixed settings (`steps` 8, `cfg` 1, `inputVideoStrength` 1) must be as they are; what this worker does not do is refused. Effects: fail, read, write, console, io. ### `serveVideos` ```gizlang function serveVideos(sys: System, m: VideoModel, host: string, port: i64, state: string, capacity: i64 = 4): i64 throws string ``` Serves video model `m` as giz's inference worker (gizlang:worker, protocol v1) on host:port, its jobs under `state`: the one model `m.name`, operation video.generate, `capacity` jobs waiting. It returns when the worker stops. Effects: fail, read, write, console, io. ### `serve` ```gizlang function serve(sys: System, listener: Listener, models: Vec, images: Vec = [], encoders: Vec = []): i64 throws string ``` Serves the OpenAI API with chat models `models` and image models `images` on `listener`: POST /v1/chat/completions, answered whole or as server-sent events (`stream`), POST /v1/images/generations, POST /v1/images/edits and GET /v1/models. A request names its model (`model`, with or without a provider as "org/name"; none: the first). The requests of a model run together (Engine.advance): a request that comes waits for none other, and a chat's next request reads only its new messages (Model.chatTokens); the models take steps in turn. A chat request gives `messages` (`content` as text or as text parts), max_completion_tokens or max_tokens (the context's rest when none), temperature, top_p and top_k (the model's when none), seed, stop, n (choices), the penalties, logprobs, response_format (samplingOf), and the reasoning effort (`reasoning_effort` or `reasoning.effort`): low, medium or high has a thinking model think first, and the thinking comes as `reasoning`. POST /v1/completions continues a text `prompt` the same way (max_tokens 16 when none). The usage counts the prompt tokens read from cache in prompt_tokens_details.cached_tokens. An image request is an ImageRequest (imageRequestOf: `prompt`, `size` "WIDTHxHEIGHT", `n`, `steps`, `seed`, ...; an edit also its pictures as `images` data URLs or as multipart files); its images come as WebP, PNG or JPEG (imageOutputOf) in `b64_json`, made between two steps of the chats, which wait for them. The encoders `encoders` answer POST /v1/embeddings (embeddingsAnswer) and POST /v1/rerank or /v2/rerank (rerankAnswer), also between two steps. It returns what Http.serve returns: the requests served, or the error that stopped it (as closing the listener). Make the engines with `ready`, or the first requests compile the kernels. On one GPU, make the engines of the smaller models first: an engine's KV pool takes the memory its slots can use, and the last engine all that is left. Effects: fail, read, write, console, io. ## Methods ### `ImageOutput.encode` ```gizlang function ImageOutput.encode(self: ImageOutput, image: Image.Image): bytes ``` `image` in the output's format. Effects: fail. ### `ImageModel.make` ```gizlang function ImageModel.make(self: ImageModel, sys: System, r: ImageRequest, progress: (stage: string, step: i64, total: i64) => bool): Made throws string ``` The images of request `r` (checkImage): `count` of them in turn, from the seeds seed, seed + 1 and on. `progress` is told the stage ("encoding_prompt", then "generating"), the step done and the steps in all; when it returns false `make` stops with CANCELLED. The tunings of the kernels it made are kept for the next start. Effects: fail, read, write, console, io. ### `VideoRequest.describe` ```gizlang function VideoRequest.describe(self: VideoRequest): string ``` The request in one line for the journal: its size and the features it uses (never its prompt or files). Effects: fail. ### `VideoModel.make` ```gizlang function VideoModel.make(self: VideoModel, sys: System, r: VideoRequest, progress: (stage: string, step: i64, total: i64) => bool): bytes throws string ``` The MP4 (H.264 and AAC) of request `r` (videoRequestOf). `progress` is told the stage ("encoding_prompt", "generating", "encoding_video"), the step done and the steps in all; when it returns false `make` stops with CANCELLED. Effects: fail, read, write, console, io. ## Constants ### `CANCELLED` ```gizlang const CANCELLED = "cancelled" ``` The error `make` gives when its progress callback says to stop. ### `NO_ROOM` ```gizlang const NO_ROOM = "no room for the prompt upsampler beside the model on this GPU" ``` The error `make` gives when the upsampler does not fit beside the model (the request can go without it). ### `TOO_LARGE` ```gizlang const TOO_LARGE = "too large for this GPU" ``` The start of the error `make` gives for a request that does not fit the GPU beside the model (a smaller size or fewer pictures would). ### `LTX` ```gizlang const LTX = "ltx2-5-distilled" ``` The name giz gives LTX-2.5 22B distilled, the video model.