Run AI models on your computer

One command chats with a model or makes an image. You give a model's name. The first use downloads the model from Hugging Face and converts it once. Later uses start from the converted copy in seconds and need no network. The same models serve the OpenAI API, so coding agents and other OpenAI clients can use them.

• A chat model runs on an NVIDIA or AMD GPU, or on the CPU (--cpu, slower). Every model of the catalog fits a GPU of 16 GB with room for long chats.
• An Apple GPU (Metal) is written and tested without a Mac only: no kernel has run on one yet, so its speed is not measured.
• An image model needs an NVIDIA GPU with INT8 matrix cores (sm_80 and later).
• Each example shows real output. Long output is trimmed (…).

Step 1: Chat with a model

giz ai chat qwen3.5-4b "Why is the sky blue? Two sentences."

The sky appears blue because sunlight is scattered by the Earth's atmosphere, and shorter blue wavelengths are scattered
more effectively than longer red wavelengths. This phenomenon, known as Rayleigh scattering, causes the blue light to
reach our eyes from all directions in the sky.

• The answer goes to standard output as it comes. The first use prints one line on standard error: it downloads Qwen/Qwen3.5-4B (8.8 GB) and converts it (5.3 GB). On one RTX 4080 that first run took 405 s. The next run took 5.3 s, from the start of the command to the end of the answer.
• Without a prompt, chat reads your messages line by line and keeps the chat. End it with Ctrl-D.
• --think lets a thinking model think first. The thinking goes to standard error, so the answer stays clean.
• --max N limits an answer to N tokens. --context N sets the positions of the chat (32768 by default).
• giz ai models lists the catalog and marks the models that are ready on this computer:

giz ai models

The models folder is /home/you/.cache/gizlang/huggingface. A Hugging Face repository ("owner/name") is a model too.
qwen3.5-0.8b  text   here  Qwen/Qwen3.5-0.8B                       Qwen 3.5 0.8B: a small, fast chat model
qwen3.5-2b    text         Qwen/Qwen3.5-2B                         Qwen 3.5 2B
qwen3.5-4b    text   here  Qwen/Qwen3.5-4B                         Qwen 3.5 4B: a good first model
qwen3.5-9b    text         Qwen/Qwen3.5-9B                         Qwen 3.5 9B
qwen3.5-27b   text         Qwen/Qwen3.5-27B                        Qwen 3.5 27B: the strongest in 16 GB, for coding agents
flux3-klein   image        black-forest-labs/flux-3-image-klein    FLUX.3 klein: text to image in 16 steps (gated: accept its license and set HF_TOKEN)
z-image-turbo image        Comfy-Org/z_image_turbo                 Z-Image Turbo: text to image in 8 steps
e5-small      embed        intfloat/multilingual-e5-small          E5 small: multilingual embeddings (384 values)
minilm-rank   rank         cross-encoder/ms-marco-MiniLM-L6-v2     MS MARCO MiniLM: an English reranker

Step 2: Make an image

FLUX.3 klein is a gated repository. Before its first use:

1. Sign in at Hugging Face and accept the license at black-forest-labs/flux-3-image-klein.
2. Make a token with read access at huggingface.co/settings/tokens.
3. Set it: export HF_TOKEN=hf_….

giz ai image "a red fox in fresh snow, morning light" --seed 42 --out fox.png

fox.png: 1024x1024 in 2.381 s

• The first use downloads 19 GB and converts the text encoder (3.8 GB). On one RTX 4080 SUPER that first run took 625 s. The first image of a size also tunes the GPU kernels for it (8.6 s for 1024x1024). Later images take 2.4 s, and the same seed gives the same image.
• --size WIDTHxHEIGHT sets the size in pixels (multiples of 16, 1024x1024 by default). --steps N sets the steps (by default the model's: 16 for klein, 8 for Z-Image Turbo, as each is distilled for).
• --seed N gives the same noise as ComfyUI's for that seed, so the same seed and prompt give ComfyUI's image.
• --model z-image-turbo makes the image with Z-Image Turbo instead (not gated). It makes images from text only. Its text encoder comes from Tongyi-MAI/Z-Image-Turbo and is converted at its first use.
• --ref PICTURE edits a picture instead (give it up to four times: the image has the first one's aspect, at 1 megapixel unless --size is given). A JPEG, PNG or WebP of any size is read, at 2048 pixels on its longer side.
• --count N makes N images in turn from seeds SEED, SEED + 1 and on (fox-1.png and on). --out picks the format by its extension: .webp (the default, image.webp), .png, or .jpg for a JPEG.
• --upsample has BFL's prompt upsampler (Qwen3.5-9B) write the prompt out in full first. Its first use downloads and converts its repository once; it needs 6.4 GB of GPU memory beside klein, and without it the command says so.
• Without the license accepted, the download stops with a message that names HF_TOKEN and the license page.

Step 3: Serve the OpenAI API

giz ai serve qwen3.5-27b --port 8000

serving qwen3.5-27b at http://127.0.0.1:8000/v1

Any OpenAI client can use it. Give it the address and the model's name:

curl -s http://127.0.0.1:8000/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model": "qwen3.5-27b", "messages": [{"role": "user", "content": "Hello"}], "stream": true}'

• The server answers POST /v1/chat/completions (whole or streamed, with tools and reasoning_effort), POST /v1/completions (a text prompt), POST /v1/images/generations (WebP as b64_json unless output_format asks for png or jpeg) and GET /v1/models.
• A chat or completion request takes OpenAI's sampling fields: temperature, top_p, seed, n (up to 16 choices), stop, presence_penalty, frequency_penalty and logprobs (a chat: true with top_logprobs up to 20; a completion: the count). top_k and repetition_penalty are vLLM's. response_format {"type": "json_object"} or {"type": "json_schema", "json_schema": {"schema": ...}} lets the model write only JSON of that schema (after its thinking): gizml:grammar masks each token that cannot continue it. A schema's type, properties, required, additionalProperties (false when not given), items, minItems, maxItems, enum, const, anyOf, oneOf and $ref hold; pattern, format and lengths do not. A request with these fields makes no MTP drafts.
• A chat message's content parts image_url and video_url (a data URL each: data:...;base64,...) show a picture or a video to a model with a vision tower (Qwen3.5, Qwen3-VL: visual.safetensors in its folder). A picture keeps its aspect in sides of multiples of 32, from 65,536 to 1,048,576 pixels (up to 1024 prompt tokens). A video gives 2 frames a second, at most 32 over the whole video, each at most 200,704 pixels; ffmpeg on the PATH decodes it. Another model refuses such a request.
• POST /v1/images/edits edits pictures, as OpenAI's: JSON with images (data URLs), or a multipart form with the pictures as files. n, size, output_format and output_compression are OpenAI's names; steps, seed, upsample and stepCache are gizlang's.
• POST /v1/embeddings gives the embeddings of an embed model: input is a text or a list of texts, and encoding_format is float (the default) or base64 (the f32 values, little-endian, as the OpenAI clients ask).
• POST /v1/rerank (and /v2/rerank, the Jina and Cohere API) ranks documents for a query with a rank model: results come by relevance_score, the highest first. top_n keeps that many, and return_documents: false leaves out the texts.
• Name more than one model to serve them together, for example giz ai serve qwen3.5-4b flux3-klein e5-small. The first chat model is the default. An image or an encoding is made between two steps of the chats.
• --slots N sets the chats that run together (4 by default). A chat's next request reads only its new messages: the engine keeps the rest. --host 0.0.0.0 listens on every address.
• For a coding agent, set its OpenAI base URL to http://127.0.0.1:8000/v1 and its model to the name you served.

Step 4: Add a model

Any Hugging Face repository of a family that gizml:llm reads is a model: Qwen 3.5, Qwen 3, the text model of Qwen3-VL, Qwen 2, Llama and Mistral. Give its id as the name:

giz ai chat Qwen/Qwen3-8B "Hello"

• Such a model takes 4 bits for each linear layer and 6 bits for the output layer.
• A name of the catalog is one line in catalog() of ml/ai.giz: its name, repository, bits and a short text.
• A new family is one recipe in gizml:llm: how its config.json reads, and the names of its tensors.

Step 5: Use the models in a program

gizml:ai gives the same models to a program. Ai.models opens the models folder, Ai.text loads a chat model by name, Ai.ready makes its engine, and Ai.reply gives an answer piece by piece. Ai.image loads an image model.

import {System} from "gizlang:system";
import * as Image from "gizlang:image";
import {Message} from "gizml:llm";
import * as Ai from "gizml:ai";

function main(sys: System): void {
    const hub = Ai.models(sys)!;
    const m = Ai.text(hub, "qwen3.5-4b")!;
    let e = Ai.ready(sys, m, 1, 8192)!;
    const asked: Vec<Message> = [{ role: "user", content: "Name three rivers." }];
    const answer = Ai.reply(sys, e, asked, Ai.ask(), (p: Ai.Piece): bool => true)!;
    print(answer.text);
    let painter = Ai.image(hub)!;
    let r = Ai.imageRequest("a red fox in fresh snow");
    r.seed = 42;
    const made = painter.make(sys, r, (stage: string, step: i64, total: i64): bool => true)!;
    sys.replaceFile("fox.png", Image.png(made.images[0]))!;
}

• Ai.ImageRequest is one request for any image model, whether it comes from the command line, the server or a worker: the prompt, pictures to edit, the size, steps, seed, count, and the options below. Ai.checkImage says whether a request is within the bounds; ImageModel.make makes its images and tells its progress, and stops when the callback returns false.
• Ai.serve(sys, listener, models, images, encoders) is the server of Step 3. Ai.encoder(hub, name) loads an embed or rank model for it.
• Ai.serveImages(sys, model, host, port, state) serves an image model as giz's inference worker (Step 6).
• Tensors, GPU kernels and AI models shows the parts under these functions: gizml:llm's convert, load and Engine, and gizml:diffusion's FLUX.3 and MiniMax H3.

Where the files are

The models folder is $GIZLANG_CACHE/huggingface, else ~/.cache/gizlang/huggingface.

• A download is checked against the SHA-256 that Hugging Face states, and a stopped download leaves no file.
• To remove a model, remove its folders. Its next use downloads it again.
• HF_ENDPOINT names a mirror of Hugging Face, for example a server in your network.

Step 6: Serve an image model as a worker

For a service that takes long jobs, GizStack serves an image model as its inference worker: jobs are queued, report their progress and can be cancelled, and each result is kept until the caller acknowledges it (gizlang:worker, protocol v1).

giz ai worker flux3-klein --state /var/lib/flux --port 5079

• POST /v1/jobs takes {"operation": "image.generate", "model": "flux3-klein", "input": {...}} as JSON, or as the form field request with the pictures as files named in referenceImages ({"$file": "r0"}). The input is the image request of Step 5: prompt, referenceImages (up to four: none is text to image), width and height or megapixels, steps, seed (-1: from the clock), count, stepCache, upsample, and outputFormat (webp by default, png, jpeg) with outputQuality (90).
• --id ID serves the model under another model id, the one the callers already send (for example z_image_turbo). numberResults is a name for count too.
• GET /v1/jobs/ID gives the status, the progress (encoding_prompt, then generating with its step) and the outputs: /files/ID/image-1.webp and on, and prompt.txt when the upsampler wrote the prompt. DELETE cancels a job between two steps. A job that the input makes impossible fails with WORKER_INVALID_INPUT and a message to show; one that does not fit the GPU's free memory now fails with WORKER_NO_MEMORY (status 500: the worker's capacity).
• The jobs and their files stay under --state: a restart keeps the queued ones and fails a job that was running with WORKER_INTERRUPTED.