# Run AI models on your computer One command chats with a model or makes an image. You give a model's name. The first use downloads the model from Hugging Face and converts it once. Later uses start from the converted copy in seconds and need no network. The same models serve the OpenAI API, so coding agents and other OpenAI clients can use them. - A chat model runs on an NVIDIA or AMD GPU, or on the CPU (`--cpu`, slower). Every model of the catalog fits a GPU of 16 GB with room for long chats. - An Apple GPU (Metal) is written and tested without a Mac only: no kernel has run on one yet, so its speed is not measured. - An image model needs an NVIDIA GPU with INT8 matrix cores (sm_80 and later). - Each example shows real output. Long output is trimmed (`…`). ## Step 1: Chat with a model ```sh giz ai chat qwen3.5-4b "Why is the sky blue? Two sentences." ``` ```text The sky appears blue because sunlight is scattered by the Earth's atmosphere, and shorter blue wavelengths are scattered more effectively than longer red wavelengths. This phenomenon, known as Rayleigh scattering, causes the blue light to reach our eyes from all directions in the sky. ``` - The answer goes to standard output as it comes. The first use prints one line on standard error: it downloads `Qwen/Qwen3.5-4B` (8.8 GB) and converts it (5.3 GB). On one RTX 4080 that first run took 405 s. The next run took 5.3 s, from the start of the command to the end of the answer. - Without a prompt, `chat` reads your messages line by line and keeps the chat. End it with Ctrl-D. - `--think` lets a thinking model think first. The thinking goes to standard error, so the answer stays clean. - `--max N` limits an answer to N tokens. `--context N` sets the positions of the chat (32768 by default). - `giz ai models` lists the catalog and marks the models that are ready on this computer: ```sh giz ai models ``` ```text The models folder is /home/you/.cache/gizlang/huggingface. A Hugging Face repository ("owner/name") is a model too. qwen3.5-0.8b text here Qwen/Qwen3.5-0.8B Qwen 3.5 0.8B: a small, fast chat model qwen3.5-2b text Qwen/Qwen3.5-2B Qwen 3.5 2B qwen3.5-4b text here Qwen/Qwen3.5-4B Qwen 3.5 4B: a good first model qwen3.5-9b text Qwen/Qwen3.5-9B Qwen 3.5 9B qwen3.5-27b text Qwen/Qwen3.5-27B Qwen 3.5 27B: the strongest in 16 GB, for coding agents flux3-klein image black-forest-labs/flux-3-image-klein FLUX.3 klein: text to image in 16 steps (gated: accept its license and set HF_TOKEN) z-image-turbo image Comfy-Org/z_image_turbo Z-Image Turbo: text to image in 8 steps e5-small embed intfloat/multilingual-e5-small E5 small: multilingual embeddings (384 values) minilm-rank rank cross-encoder/ms-marco-MiniLM-L6-v2 MS MARCO MiniLM: an English reranker ``` ## Step 2: Make an image FLUX.3 klein is a gated repository. Before its first use: 1. Sign in at Hugging Face and accept the license at [black-forest-labs/flux-3-image-klein](https://huggingface.co/black-forest-labs/flux-3-image-klein). 2. Make a token with read access at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens). 3. Set it: `export HF_TOKEN=hf_…`. ```sh giz ai image "a red fox in fresh snow, morning light" --seed 42 --out fox.png ``` ```text fox.png: 1024x1024 in 2.381 s ``` - The first use downloads 19 GB and converts the text encoder (3.8 GB). On one RTX 4080 SUPER that first run took 625 s. The first image of a size also tunes the GPU kernels for it (8.6 s for 1024x1024). Later images take 2.4 s, and the same seed gives the same image. - `--size WIDTHxHEIGHT` sets the size in pixels (multiples of 16, 1024x1024 by default). `--steps N` sets the steps (by default the model's: 16 for klein, 8 for Z-Image Turbo, as each is distilled for). - `--seed N` gives the same noise as ComfyUI's for that seed, so the same seed and prompt give ComfyUI's image. - `--model z-image-turbo` makes the image with Z-Image Turbo instead (not gated). It makes images from text only. Its text encoder comes from Tongyi-MAI/Z-Image-Turbo and is converted at its first use. - `--ref PICTURE` edits a picture instead (give it up to four times: the image has the first one's aspect, at 1 megapixel unless `--size` is given). A JPEG, PNG or WebP of any size is read, at 2048 pixels on its longer side. - `--count N` makes N images in turn from seeds SEED, SEED + 1 and on (`fox-1.png` and on). `--out` picks the format by its extension: `.webp` (the default, `image.webp`), `.png`, or `.jpg` for a JPEG. - `--upsample` has BFL's prompt upsampler (Qwen3.5-9B) write the prompt out in full first. Its first use downloads and converts its repository once; it needs 6.4 GB of GPU memory beside klein, and without it the command says so. - Without the license accepted, the download stops with a message that names `HF_TOKEN` and the license page. ## Step 3: Serve the OpenAI API ```sh giz ai serve qwen3.5-27b --port 8000 ``` ```text serving qwen3.5-27b at http://127.0.0.1:8000/v1 ``` Any OpenAI client can use it. Give it the address and the model's name: ```sh curl -s http://127.0.0.1:8000/v1/chat/completions -H 'content-type: application/json' \ -d '{"model": "qwen3.5-27b", "messages": [{"role": "user", "content": "Hello"}], "stream": true}' ``` - The server answers `POST /v1/chat/completions` (whole or streamed, with tools and `reasoning_effort`), `POST /v1/completions` (a text `prompt`), `POST /v1/images/generations` (WebP as `b64_json` unless `output_format` asks for `png` or `jpeg`) and `GET /v1/models`. - A chat or completion request takes OpenAI's sampling fields: `temperature`, `top_p`, `seed`, `n` (up to 16 choices), `stop`, `presence_penalty`, `frequency_penalty` and `logprobs` (a chat: `true` with `top_logprobs` up to 20; a completion: the count). `top_k` and `repetition_penalty` are vLLM's. `response_format` `{"type": "json_object"}` or `{"type": "json_schema", "json_schema": {"schema": ...}}` lets the model write only JSON of that schema (after its thinking): `gizml:grammar` masks each token that cannot continue it. A schema's `type`, `properties`, `required`, `additionalProperties` (false when not given), `items`, `minItems`, `maxItems`, `enum`, `const`, `anyOf`, `oneOf` and `$ref` hold; `pattern`, `format` and lengths do not. A request with these fields makes no MTP drafts. - A chat message's content parts `image_url` and `video_url` (a data URL each: `data:...;base64,...`) show a picture or a video to a model with a vision tower (Qwen3.5, Qwen3-VL: `visual.safetensors` in its folder). A picture keeps its aspect in sides of multiples of 32, from 65,536 to 1,048,576 pixels (up to 1024 prompt tokens). A video gives 2 frames a second, at most 32 over the whole video, each at most 200,704 pixels; `ffmpeg` on the PATH decodes it. Another model refuses such a request. - `POST /v1/images/edits` edits pictures, as OpenAI's: JSON with `images` (data URLs), or a multipart form with the pictures as files. `n`, `size`, `output_format` and `output_compression` are OpenAI's names; `steps`, `seed`, `upsample` and `stepCache` are gizlang's. - `POST /v1/embeddings` gives the embeddings of an `embed` model: `input` is a text or a list of texts, and `encoding_format` is `float` (the default) or `base64` (the f32 values, little-endian, as the OpenAI clients ask). - `POST /v1/rerank` (and `/v2/rerank`, the Jina and Cohere API) ranks `documents` for a `query` with a `rank` model: `results` come by `relevance_score`, the highest first. `top_n` keeps that many, and `return_documents: false` leaves out the texts. - Name more than one model to serve them together, for example `giz ai serve qwen3.5-4b flux3-klein e5-small`. The first chat model is the default. An image or an encoding is made between two steps of the chats. - `--slots N` sets the chats that run together (4 by default). A chat's next request reads only its new messages: the engine keeps the rest. `--host 0.0.0.0` listens on every address. - For a coding agent, set its OpenAI base URL to `http://127.0.0.1:8000/v1` and its model to the name you served. ## Step 4: Add a model Any Hugging Face repository of a family that `gizml:llm` reads is a model: Qwen 3.5, Qwen 3, the text model of Qwen3-VL, Qwen 2, Llama and Mistral. Give its id as the name: ```sh giz ai chat Qwen/Qwen3-8B "Hello" ``` - Such a model takes 4 bits for each linear layer and 6 bits for the output layer. - A name of the catalog is one line in `catalog()` of `ml/ai.giz`: its name, repository, bits and a short text. - A new family is one recipe in `gizml:llm`: how its `config.json` reads, and the names of its tensors. ## Step 5: Use the models in a program `gizml:ai` gives the same models to a program. `Ai.models` opens the models folder, `Ai.text` loads a chat model by name, `Ai.ready` makes its engine, and `Ai.reply` gives an answer piece by piece. `Ai.image` loads an image model. ```gizlang import {System} from "gizlang:system"; import * as Image from "gizlang:image"; import {Message} from "gizml:llm"; import * as Ai from "gizml:ai"; function main(sys: System): void { const hub = Ai.models(sys)!; const m = Ai.text(hub, "qwen3.5-4b")!; let e = Ai.ready(sys, m, 1, 8192)!; const asked: Vec = [{ role: "user", content: "Name three rivers." }]; const answer = Ai.reply(sys, e, asked, Ai.ask(), (p: Ai.Piece): bool => true)!; print(answer.text); let painter = Ai.image(hub)!; let r = Ai.imageRequest("a red fox in fresh snow"); r.seed = 42; const made = painter.make(sys, r, (stage: string, step: i64, total: i64): bool => true)!; sys.replaceFile("fox.png", Image.png(made.images[0]))!; } ``` - `Ai.ImageRequest` is one request for any image model, whether it comes from the command line, the server or a worker: the prompt, `pictures` to edit, the size, `steps`, `seed`, `count`, and the options below. `Ai.checkImage` says whether a request is within the bounds; `ImageModel.make` makes its images and tells its progress, and stops when the callback returns false. - `Ai.serve(sys, listener, models, images, encoders)` is the server of Step 3. `Ai.encoder(hub, name)` loads an `embed` or `rank` model for it. - `Ai.serveImages(sys, model, host, port, state)` serves an image model as giz's inference worker (Step 6). - [Tensors, GPU kernels and AI models](compute.md) shows the parts under these functions: `gizml:llm`'s `convert`, `load` and `Engine`, and `gizml:diffusion`'s FLUX.3 and MiniMax H3. ## Where the files are The models folder is `$GIZLANG_CACHE/huggingface`, else `~/.cache/gizlang/huggingface`. | Path | What it holds | | --- | --- | | `OWNER/NAME/` | The downloaded files: JSON, safetensors, tokenizer and template files. Never code. | | `.pins/OWNER/NAME.pin` | The commit of the downloads and the SHA-256 of each file. A later download of the same model takes the same commit. | | `.gizlang-llm/OWNER/NAME/qL-H/` | The model converted with L bits for its layers and H for its output layer, and its kernel tunings. | - A download is checked against the SHA-256 that Hugging Face states, and a stopped download leaves no file. - To remove a model, remove its folders. Its next use downloads it again. - `HF_ENDPOINT` names a mirror of Hugging Face, for example a server in your network. ## Step 6: Serve an image model as a worker For a service that takes long jobs, GizStack serves an image model as its inference worker: jobs are queued, report their progress and can be cancelled, and each result is kept until the caller acknowledges it (`gizlang:worker`, protocol v1). ```sh giz ai worker flux3-klein --state /var/lib/flux --port 5079 ``` - `POST /v1/jobs` takes `{"operation": "image.generate", "model": "flux3-klein", "input": {...}}` as JSON, or as the form field `request` with the pictures as files named in `referenceImages` (`{"$file": "r0"}`). The input is the image request of Step 5: `prompt`, `referenceImages` (up to four: none is text to image), `width` and `height` or `megapixels`, `steps`, `seed` (-1: from the clock), `count`, `stepCache`, `upsample`, and `outputFormat` (`webp` by default, `png`, `jpeg`) with `outputQuality` (90). - `--id ID` serves the model under another model id, the one the callers already send (for example `z_image_turbo`). `numberResults` is a name for `count` too. - `GET /v1/jobs/ID` gives the status, the progress (`encoding_prompt`, then `generating` with its step) and the outputs: `/files/ID/image-1.webp` and on, and `prompt.txt` when the upsampler wrote the prompt. `DELETE` cancels a job between two steps. A job that the input makes impossible fails with `WORKER_INVALID_INPUT` and a message to show; one that does not fit the GPU's free memory now fails with `WORKER_NO_MEMORY` (status 500: the worker's capacity). - The jobs and their files stay under `--state`: a restart keeps the queued ones and fails a job that was running with `WORKER_INTERRUPTED`.