# gizai:tasks/image the image task. A model family that makes images gives a Painter (gizai:family's `image`); an ImageModel is a painter on a GPU with what a request needs beside it. This module owns what is the same for every image model: the requests as giz's workers and the OpenAI images API take them, their checks, the GPU room they need, BFL's prompt upsampler, and a worker of one model (serveImages). ```gizlang import * as Image from "gizai:tasks/image"; ``` ## Types ### `Painter` ```gizlang type Painter = { settled: (r: ImageRequest) => ImageRequest; takes: (r: ImageRequest) => Result; paint: (sys: System, r: ImageRequest, i: i64, prompt: string, progress: (step: i64, total: i64) => bool, fits: (rows: i64, width: i64, height: i64) => Result, stepped: () => i64) => Result; } ``` What a model family does with an image request (gizai:family's `image` gives it, its model loaded on a GPU): - `settled`: the request with the model's defaults in place of 0 (its steps, a workflow's size). - `takes`: whether the model takes the settled request, else why not. - `paint`: image `i` of the settled request (its seed r.seed + i) of `prompt`: the model's text rows, then its steps (`progress` hears the step done and the steps), then the decode. It calls `fits` with the rows of the joint sequence and the image's size before its steps (an error stops it), and `stepped` after them. - `settled`: `(r: M57.ImageRequest) => M57.ImageRequest` - `takes`: `(r: M57.ImageRequest) => _gizlang_sdk_core.Result` - `paint`: `(sys: gizlang_system.System, r: M57.ImageRequest, i: i64, prompt: string, progress: (step: i64, total: i64) => bool, fits: (rows: i64, width: i64, height: i64) => _gizlang_sdk_core.Result, stepped: () => i64) => _gizlang_sdk_core.Result` ### `Painted` ```gizlang type Painted = { image: Image.Image; rows: i64; stepped: i64; } ``` What `paint` made: the image, the rows of the joint sequence of its request, and what `stepped` gave. - `image`: `gizlang_image.Image` - `rows`: `i64` - `stepped`: `i64` ### `ImageModel` ```gizlang type ImageModel = { name: string; hub: Hub.Hub; gpu: Gpu; painter: Painter; tunings: string; need: ImageNeed; upsampler: Ref; } ``` An image model on a GPU: its painter, the folder of its kernels' tunings, what a request needs beside the model (measured at load), and what the upsampler took at its last use. - `name`: `string` - `hub`: `gizlang_hub.Hub` - `gpu`: `M10.Gpu` - `painter`: `M57.Painter` - `tunings`: `string` - `need`: `M57.ImageNeed` - `upsampler`: `Ref` ### `ImageRequest` ```gizlang type ImageRequest = { prompt: string; pictures: Vec; width: i64; height: i64; megapixels: f64; steps: i64; seed: i64; count: i64; cache: f64; refresh: i64; upsample: bool; workflow: string; mask: Vec; margins: Vec; negative: string; guide: Vec; } ``` What is asked of an image model: the `prompt`; the `pictures` to edit (at most 4; none: text to image); the size, `width` × `height` pixels (multiples of 16, at most 4 megapixels; 0: 1024 × 1024 for text to image, else the first picture's aspect at `megapixels`); the `steps` (0: the model's); `seed`: the first of `count` images' noise seeds (seed, seed + 1, ...: the same seed gives ComfyUI's image, Vae.fluxNoise); `cache`, the step cache's limit (0 to 0.5: more is faster and a little less exact); `refresh`, the steps between the pictures' key and value updates (0: never needed); and `upsample`, the prompt written out in full by BFL's upsampler first. - `prompt`: `string` - `pictures`: `Vec` - `width`: `i64` - `height`: `i64` - `megapixels`: `f64` - `steps`: `i64` - `seed`: `i64` - `count`: `i64` - `cache`: `f64` - `refresh`: `i64` - `upsample`: `bool` - `workflow`: `string` - `mask`: `Vec` - `margins`: `Vec` - `negative`: `string` - `guide`: `Vec` ### `ImageOutput` ```gizlang type ImageOutput = { extension: string; quality: i64; } ``` How images are written: the file extension ("webp", "png" or "jpg") and the WebP or JPEG quality. - `extension`: `string` - `quality`: `i64` ### `ImageNeed` ```gizlang type ImageNeed = { fixed: i64; perRow: i64; perToken: i64; } ``` What a request needs beside the model, measured at load (`measure`): for its steps, `fixed` bytes and `perRow` bytes for each row of its joint sequence (the image's tokens, the text's and the pictures'); for its decode, `perToken` bytes for each image token the decoder holds at once (at most Vae.IMAGE_WHOLE: a larger image decodes in tiles). - `fixed`: `i64` - `perRow`: `i64` - `perToken`: `i64` ### `Made` ```gizlang type Made = { images: Vec; prompt: string; } ``` What `make` made: the images, and the prompt they were made from (the upsampler's when `upsample`). - `images`: `Vec` - `prompt`: `string` ## Functions ### `imageModel` ```gizlang function imageModel(hub: Hub.Hub, name: string, gpu: Gpu, painter: Painter, tunings: string): ImageModel throws string ``` The image model `name` of `painter` on `gpu`: what a request needs beside it measured, and the kernels' tunings kept in `tunings` (loaded before the load, saved after the measure). Effects: fail, read, write, console, io. ### `imageRequest` ```gizlang function imageRequest(prompt: string): ImageRequest ``` A text-to-image request for `prompt` with every other setting at its default. Effects: fail. ### `checkImage` ```gizlang function checkImage(r: ImageRequest): bool throws string ``` Whether `r` is a request an image model takes: the one place that says what is too big, too many or out of range. Effects: fail. ### `imageRequestOf` ```gizlang function imageRequestOf(j: JsonValue, pictures: Vec, seed0: i64): ImageRequest throws string ``` The request of a JSON object (or a form's text fields) and the pictures that came with it: `prompt`, `size` ("WIDTHxHEIGHT") or `width` and `height`, `megapixels`, `steps`, `seed` (`seed0` when none), `count`, `n` or `numberResults`, `stepCache`, `refresh` and `upsample`. It is not checked (checkImage). Effects: fail. ### `outpaintMargins` ```gizlang function outpaintMargins(percents: string, ratio: string, width: i64, height: i64): Vec throws string ``` The outpaint margins of a `width` x `height` picture in pixels (top, bottom, left, right): `percents`, "T B L R" of the height and width ("": 10 each, at least one above 0); `ratio`, WIDTH:HEIGHT ("": none), widens the two smaller margins to it. Each is a multiple of 16, at least 16 where it is above 0. Effects: fail. ### `imageOutputOf` ```gizlang function imageOutputOf(j: JsonValue): ImageOutput throws string ``` The output of a request's `outputFormat` (or OpenAI's `output_format`): webp (the default, a tenth of a PNG's bytes), png or jpeg, at `outputQuality` (or `output_compression`; 90). Effects: fail. ### `serveImages` ```gizlang function serveImages(sys: System, name: string, place: Hosting.Place, load: (gpu: Gpu) => Result, capacity: i64 = 4): i64 throws string ``` Serves image model `name` as giz's inference worker at `place`: operation image.generate, `capacity` jobs held; `load` makes the model on the host's GPU. It returns when the worker stops. Effects: fail, read, write, console, io. ## Methods ### `ImageOutput.encode` ```gizlang function ImageOutput.encode(self: ImageOutput, image: Image.Image): bytes ``` `image` in the output's format. Effects: fail. ### `ImageModel.caption` ```gizlang function ImageModel.caption(self: ImageModel, sys: System, r: ImageRequest): Result ``` The fuller prompt that the upsampler writes for `r`. It loads beside the model on the model's Gpu, so the model's pool of dropped buffers goes back first, and its own buffers go back after; it needs what it took at its last use (Gpu.peak), or before that its files' bytes, and where that does not fit this is NO_ROOM. Effects: fail, read, write, console, io. ### `ImageModel.probe` ```gizlang function ImageModel.probe(self: ImageModel, sys: System, side: i64): Result, string> ``` Of one step and the decode of a `side` × `side` text-to-image job: the rows, the device bytes (Gpu.peak) of the step, the device bytes of the decode, and the image's tokens. Effects: fail, read, write, console, io. ### `ImageModel.takes` ```gizlang function ImageModel.takes(self: ImageModel, r: ImageRequest): bool throws string ``` Whether this model takes `r` as its painter settles it. Effects: fail, read, write, console, io. ### `ImageModel.paint` ```gizlang function ImageModel.paint(self: ImageModel, sys: System, r: ImageRequest, i: i64, progress: (step: i64, total: i64) => bool, probe: bool = false, text: string = ""): Painted throws string ``` Image `i` of request `r` (settled) of the prompt `text` ("": r.prompt). It fails where the request does not fit the GPU beside the model (fits), but for the `probe` that measures that. Effects: fail, read, write, console, io. ### `ImageModel.stepped` ```gizlang function ImageModel.stepped(self: ImageModel, probe: bool): i64 ``` For a `probe`, the device peak of the steps just run; the pool is then trimmed and the peak reset, so that the decode's peak counts only what it holds. Else 0. Effects: fail, read, write, io. ### `ImageModel.measure` ```gizlang function ImageModel.measure(self: ImageModel, sys: System): ImageNeed throws string ``` What a request needs on this GPU, measured: a 512² and a 1024² job; the bytes of a step for each row between them, and the bytes of the 1024² decode for each token. Effects: fail, read, write, console, io. ### `ImageModel.fits` ```gizlang function ImageModel.fits(self: ImageModel, rows: i64, width: i64, height: i64, pictures: i64): Result ``` Whether a job of `rows` (a `width` × `height` image with `pictures`) fits the GPU beside the model now: the more of its steps' need and its decode's (ImageNeed; Gpu.fits). A job that does not would stop the program in Gpu.bytes; it is refused with what to change instead. Effects: fail, read, write, io. ### `ImageModel.make` ```gizlang function ImageModel.make(self: ImageModel, sys: System, r0: ImageRequest, progress: (stage: string, step: i64, total: i64) => bool): Made throws string ``` The images of request `r` (checkImage): `count` of them in turn, from the seeds seed, seed + 1 and on. `progress` is told the stage ("encoding_prompt", then "generating"), the step done and the steps in all; when it returns false `make` stops with CANCELLED. The tunings of the kernels it made are kept for the next start. Effects: fail, read, write, console, io. ## Constants ### `CANCELLED` ```gizlang const CANCELLED = "cancelled" ``` The error `make` gives when its progress callback says to stop. ### `NO_ROOM` ```gizlang const NO_ROOM = "no room for the prompt upsampler beside the model on this GPU" ``` The error `make` gives when the upsampler does not fit beside the model (the request can go without it).