# gizai:tasks/image

the image task. A model family that makes images gives a Painter (gizai:family's `image`); an ImageModel is a painter on a GPU with what a request needs beside it. This module owns what is the same for every image model: the requests as giz's workers and the OpenAI images API take them, their checks, the GPU room they need, BFL's prompt upsampler, and a worker of one model (serveImages).

```gizlang
import * as Image from "gizai:tasks/image";
```

## Types

### `Painter`

```gizlang
type Painter = { settled: (r: ImageRequest) => ImageRequest; takes: (r: ImageRequest) => Result<bool, string>; paint: (sys: System, r: ImageRequest, i: i64, prompt: string, progress: (step: i64, total: i64) => bool, fits: (rows: i64, width: i64, height: i64) => Result<bool, string>, stepped: () => i64) => Result<Painted, string>; }
```

What a model family does with an image request (gizai:family's `image` gives it, its model loaded on a GPU):
- `settled`: the request with the model's defaults in place of 0 (its steps, a workflow's size).
- `takes`: whether the model takes the settled request, else why not.
- `paint`: image `i` of the settled request (its seed r.seed + i) of `prompt`: the model's text rows, then its steps
  (`progress` hears the step done and the steps), then the decode. It calls `fits` with the rows of the joint
  sequence and the image's size before its steps (an error stops it), and `stepped` after them.

- `settled`: `(r: M57.ImageRequest) => M57.ImageRequest`
- `takes`: `(r: M57.ImageRequest) => _gizlang_sdk_core.Result<bool, string>`
- `paint`: `(sys: gizlang_system.System, r: M57.ImageRequest, i: i64, prompt: string, progress: (step: i64, total: i64) => bool, fits: (rows: i64, width: i64, height: i64) => _gizlang_sdk_core.Result<bool, string>, stepped: () => i64) => _gizlang_sdk_core.Result<M57.Painted, string>`

### `Painted`

```gizlang
type Painted = { image: Image.Image; rows: i64; stepped: i64; }
```

What `paint` made: the image, the rows of the joint sequence of its request, and what `stepped` gave.

- `image`: `gizlang_image.Image`
- `rows`: `i64`
- `stepped`: `i64`

### `ImageModel`

```gizlang
type ImageModel = { name: string; hub: Hub.Hub; gpu: Gpu; painter: Painter; tunings: string; need: ImageNeed; upsampler: Ref<i64>; }
```

An image model on a GPU: its painter, the folder of its kernels' tunings, what a request needs beside the model
(measured at load), and what the upsampler took at its last use.

- `name`: `string`
- `hub`: `gizlang_hub.Hub`
- `gpu`: `M10.Gpu`
- `painter`: `M57.Painter`
- `tunings`: `string`
- `need`: `M57.ImageNeed`
- `upsampler`: `Ref<i64>`

### `ImageRequest`

```gizlang
type ImageRequest = { prompt: string; pictures: Vec<Image.Image>; width: i64; height: i64; megapixels: f64; steps: i64; seed: i64; count: i64; cache: f64; refresh: i64; upsample: bool; workflow: string; mask: Vec<Image.Image>; margins: Vec<i64>; negative: string; guide: Vec<f64>; }
```

What is asked of an image model: the `prompt`; the `pictures` to edit (at most 4; none: text to image); the size,
`width` × `height` pixels (multiples of 16, at most 4 megapixels; 0: 1024 × 1024 for text to image, else the first
picture's aspect at `megapixels`); the `steps` (0: the model's); `seed`: the first of `count` images' noise
seeds (seed, seed + 1, ...: the same seed gives ComfyUI's image, Vae.fluxNoise); `cache`, the step cache's limit
(0 to 0.5: more is faster and a little less exact); `refresh`, the steps between the pictures' key and value updates
(0: never needed); and `upsample`, the prompt written out in full by BFL's upsampler first.

- `prompt`: `string`
- `pictures`: `Vec<gizlang_image.Image>`
- `width`: `i64`
- `height`: `i64`
- `megapixels`: `f64`
- `steps`: `i64`
- `seed`: `i64`
- `count`: `i64`
- `cache`: `f64`
- `refresh`: `i64`
- `upsample`: `bool`
- `workflow`: `string`
- `mask`: `Vec<gizlang_image.Image>`
- `margins`: `Vec<i64>`
- `negative`: `string`
- `guide`: `Vec<f64>`

### `ImageOutput`

```gizlang
type ImageOutput = { extension: string; quality: i64; }
```

How images are written: the file extension ("webp", "png" or "jpg") and the WebP or JPEG quality.

- `extension`: `string`
- `quality`: `i64`

### `ImageNeed`

```gizlang
type ImageNeed = { fixed: i64; perRow: i64; perToken: i64; }
```

What a request needs beside the model, measured at load (`measure`): for its steps, `fixed` bytes and `perRow` bytes
for each row of its joint sequence (the image's tokens, the text's and the pictures'); for its decode, `perToken`
bytes for each image token the decoder holds at once (at most Vae.IMAGE_WHOLE: a larger image decodes in tiles).

- `fixed`: `i64`
- `perRow`: `i64`
- `perToken`: `i64`

### `Made`

```gizlang
type Made = { images: Vec<Image.Image>; prompt: string; }
```

What `make` made: the images, and the prompt they were made from (the upsampler's when `upsample`).

- `images`: `Vec<gizlang_image.Image>`
- `prompt`: `string`

## Functions

### `imageModel`

```gizlang
function imageModel(hub: Hub.Hub, name: string, gpu: Gpu, painter: Painter, tunings: string): ImageModel throws string
```

The image model `name` of `painter` on `gpu`: what a request needs beside it measured, and the kernels' tunings
kept in `tunings` (loaded before the load, saved after the measure).

Effects: fail, read, write, console, io.

### `imageRequest`

```gizlang
function imageRequest(prompt: string): ImageRequest
```

A text-to-image request for `prompt` with every other setting at its default.

Effects: fail.

### `checkImage`

```gizlang
function checkImage(r: ImageRequest): bool throws string
```

Whether `r` is a request an image model takes: the one place that says what is too big, too many or out of range.

Effects: fail.

### `imageRequestOf`

```gizlang
function imageRequestOf(j: JsonValue, pictures: Vec<Image.Image>, seed0: i64): ImageRequest throws string
```

The request of a JSON object (or a form's text fields) and the pictures that came with it: `prompt`, `size`
("WIDTHxHEIGHT") or `width` and `height`, `megapixels`, `steps`, `seed` (`seed0` when none), `count`, `n` or `numberResults`,
`stepCache`, `refresh` and `upsample`. It is not checked (checkImage).

Effects: fail.

### `outpaintMargins`

```gizlang
function outpaintMargins(percents: string, ratio: string, width: i64, height: i64): Vec<i64> throws string
```

The outpaint margins of a `width` x `height` picture in pixels (top, bottom, left, right): `percents`, "T B L R" of
the height and width ("": 10 each, at least one above 0); `ratio`, WIDTH:HEIGHT ("": none), widens the two smaller
margins to it. Each is a multiple of 16, at least 16 where it is above 0.

Effects: fail.

### `imageOutputOf`

```gizlang
function imageOutputOf(j: JsonValue): ImageOutput throws string
```

The output of a request's `outputFormat` (or OpenAI's `output_format`): webp (the default, a tenth of a PNG's bytes),
png or jpeg, at `outputQuality` (or `output_compression`; 90).

Effects: fail.

### `serveImages`

```gizlang
function serveImages(sys: System, name: string, place: Hosting.Place, load: (gpu: Gpu) => Result<ImageModel, string>, capacity: i64 = 4): i64 throws string
```

Serves image model `name` as giz's inference worker at `place`: operation image.generate, `capacity` jobs held;
`load` makes the model on the host's GPU. It returns when the worker stops.

Effects: fail, read, write, console, io.

## Methods

### `ImageOutput.encode`

```gizlang
function ImageOutput.encode(self: ImageOutput, image: Image.Image): bytes
```

`image` in the output's format.

Effects: fail.

### `ImageModel.caption`

```gizlang
function ImageModel.caption(self: ImageModel, sys: System, r: ImageRequest): Result<string, string>
```

The fuller prompt that the upsampler writes for `r`. It loads beside the model on the model's Gpu, so the model's
pool of dropped buffers goes back first, and its own buffers go back after; it needs what it took at its last use
(Gpu.peak), or before that its files' bytes, and where that does not fit this is NO_ROOM.

Effects: fail, read, write, console, io.

### `ImageModel.probe`

```gizlang
function ImageModel.probe(self: ImageModel, sys: System, side: i64): Result<Vec<i64>, string>
```

Of one step and the decode of a `side` × `side` text-to-image job: the rows, the device bytes (Gpu.peak) of the step,
the device bytes of the decode, and the image's tokens.

Effects: fail, read, write, console, io.

### `ImageModel.takes`

```gizlang
function ImageModel.takes(self: ImageModel, r: ImageRequest): bool throws string
```

Whether this model takes `r` as its painter settles it.

Effects: fail, read, write, console, io.

### `ImageModel.paint`

```gizlang
function ImageModel.paint(self: ImageModel, sys: System, r: ImageRequest, i: i64, progress: (step: i64, total: i64) => bool, probe: bool = false, text: string = ""): Painted throws string
```

Image `i` of request `r` (settled) of the prompt `text` ("": r.prompt). It fails where the request does not fit
the GPU beside the model (fits), but for the `probe` that measures that.

Effects: fail, read, write, console, io.

### `ImageModel.stepped`

```gizlang
function ImageModel.stepped(self: ImageModel, probe: bool): i64
```

For a `probe`, the device peak of the steps just run; the pool is then trimmed and the peak reset, so that the
decode's peak counts only what it holds. Else 0.

Effects: fail, read, write, io.

### `ImageModel.measure`

```gizlang
function ImageModel.measure(self: ImageModel, sys: System): ImageNeed throws string
```

What a request needs on this GPU, measured: a 512² and a 1024² job; the bytes of a step for each row between them,
and the bytes of the 1024² decode for each token.

Effects: fail, read, write, console, io.

### `ImageModel.fits`

```gizlang
function ImageModel.fits(self: ImageModel, rows: i64, width: i64, height: i64, pictures: i64): Result<bool, string>
```

Whether a job of `rows` (a `width` × `height` image with `pictures`) fits the GPU beside the model now: the more of
its steps' need and its decode's (ImageNeed; Gpu.fits). A job that does not would stop the program in Gpu.bytes; it is refused with what to change instead.

Effects: fail, read, write, io.

### `ImageModel.make`

```gizlang
function ImageModel.make(self: ImageModel, sys: System, r0: ImageRequest, progress: (stage: string, step: i64, total: i64) => bool): Made throws string
```

The images of request `r` (checkImage): `count` of them in turn, from the seeds seed, seed + 1 and on. `progress`
is told the stage ("encoding_prompt", then "generating"), the step done and the steps in all; when it returns false
`make` stops with CANCELLED. The tunings of the kernels it made are kept for the next start.

Effects: fail, read, write, console, io.

## Constants

### `CANCELLED`

```gizlang
const CANCELLED = "cancelled"
```

The error `make` gives when its progress callback says to stop.

### `NO_ROOM`

```gizlang
const NO_ROOM = "no room for the prompt upsampler beside the model on this GPU"
```

The error `make` gives when the upsampler does not fit beside the model (the request can go without it).

gizai:tasks/image

the image task. A model family that makes images gives a Painter (gizai:family's image); an ImageModel is a painter on a GPU with what a request needs beside it. This module owns what is the same for every image model: the requests as giz's workers and the OpenAI images API take them, their checks, the GPU room they need, BFL's prompt upsampler, and a worker of one model (serveImages).

gizlang

import * as Image from "gizai:tasks/image";

Types

`Painter`

gizlang

type Painter = { settled: (r: ImageRequest) => ImageRequest; takes: (r: ImageRequest) => Result<bool, string>; paint: (sys: System, r: ImageRequest, i: i64, prompt: string, progress: (step: i64, total: i64) => bool, fits: (rows: i64, width: i64, height: i64) => Result<bool, string>, stepped: () => i64) => Result<Painted, string>; }

What a model family does with an image request (gizai:family's image gives it, its model loaded on a GPU):

settled: the request with the model's defaults in place of 0 (its steps, a workflow's size).
takes: whether the model takes the settled request, else why not.
paint: image i of the settled request (its seed r.seed + i) of prompt: the model's text rows, then its steps (progress hears the step done and the steps), then the decode. It calls fits with the rows of the joint sequence and the image's size before its steps (an error stops it), and stepped after them.
settled: (r: M57.ImageRequest) => M57.ImageRequest
takes: (r: M57.ImageRequest) => _gizlang_sdk_core.Result<bool, string>
paint: (sys: gizlang_system.System, r: M57.ImageRequest, i: i64, prompt: string, progress: (step: i64, total: i64) => bool, fits: (rows: i64, width: i64, height: i64) => _gizlang_sdk_core.Result<bool, string>, stepped: () => i64) => _gizlang_sdk_core.Result<M57.Painted, string>

`Painted`

gizlang

type Painted = { image: Image.Image; rows: i64; stepped: i64; }

What paint made: the image, the rows of the joint sequence of its request, and what stepped gave.

image: gizlang_image.Image
rows: i64
stepped: i64

`ImageModel`

gizlang

type ImageModel = { name: string; hub: Hub.Hub; gpu: Gpu; painter: Painter; tunings: string; need: ImageNeed; upsampler: Ref<i64>; }

An image model on a GPU: its painter, the folder of its kernels' tunings, what a request needs beside the model (measured at load), and what the upsampler took at its last use.

name: string
hub: gizlang_hub.Hub
gpu: M10.Gpu
painter: M57.Painter
tunings: string
need: M57.ImageNeed
upsampler: Ref<i64>

`ImageRequest`

gizlang

type ImageRequest = { prompt: string; pictures: Vec<Image.Image>; width: i64; height: i64; megapixels: f64; steps: i64; seed: i64; count: i64; cache: f64; refresh: i64; upsample: bool; workflow: string; mask: Vec<Image.Image>; margins: Vec<i64>; negative: string; guide: Vec<f64>; }

What is asked of an image model: the prompt; the pictures to edit (at most 4; none: text to image); the size, width × height pixels (multiples of 16, at most 4 megapixels; 0: 1024 × 1024 for text to image, else the first picture's aspect at megapixels); the steps (0: the model's); seed: the first of count images' noise seeds (seed, seed + 1, ...: the same seed gives ComfyUI's image, Vae.fluxNoise); cache, the step cache's limit (0 to 0.5: more is faster and a little less exact); refresh, the steps between the pictures' key and value updates (0: never needed); and upsample, the prompt written out in full by BFL's upsampler first.

prompt: string
pictures: Vec<gizlang_image.Image>
width: i64
height: i64
megapixels: f64
steps: i64
seed: i64
count: i64
cache: f64
refresh: i64
upsample: bool
workflow: string
mask: Vec<gizlang_image.Image>
margins: Vec<i64>
negative: string
guide: Vec<f64>

`ImageOutput`

gizlang

type ImageOutput = { extension: string; quality: i64; }

How images are written: the file extension ("webp", "png" or "jpg") and the WebP or JPEG quality.

extension: string
quality: i64

`ImageNeed`

gizlang

type ImageNeed = { fixed: i64; perRow: i64; perToken: i64; }

What a request needs beside the model, measured at load (measure): for its steps, fixed bytes and perRow bytes for each row of its joint sequence (the image's tokens, the text's and the pictures'); for its decode, perToken bytes for each image token the decoder holds at once (at most Vae.IMAGE_WHOLE: a larger image decodes in tiles).

fixed: i64
perRow: i64
perToken: i64

`Made`

gizlang

type Made = { images: Vec<Image.Image>; prompt: string; }

What make made: the images, and the prompt they were made from (the upsampler's when upsample).

images: Vec<gizlang_image.Image>
prompt: string

Functions

`imageModel`

gizlang

function imageModel(hub: Hub.Hub, name: string, gpu: Gpu, painter: Painter, tunings: string): ImageModel throws string

The image model name of painter on gpu: what a request needs beside it measured, and the kernels' tunings kept in tunings (loaded before the load, saved after the measure).

Effects: fail, read, write, console, io.

`imageRequest`

gizlang

function imageRequest(prompt: string): ImageRequest

A text-to-image request for prompt with every other setting at its default.

Effects: fail.

`checkImage`

gizlang

function checkImage(r: ImageRequest): bool throws string

Whether r is a request an image model takes: the one place that says what is too big, too many or out of range.

Effects: fail.

`imageRequestOf`

gizlang

function imageRequestOf(j: JsonValue, pictures: Vec<Image.Image>, seed0: i64): ImageRequest throws string

The request of a JSON object (or a form's text fields) and the pictures that came with it: prompt, size ("WIDTHxHEIGHT") or width and height, megapixels, steps, seed (seed0 when none), count, n or numberResults, stepCache, refresh and upsample. It is not checked (checkImage).

Effects: fail.

`outpaintMargins`

gizlang

function outpaintMargins(percents: string, ratio: string, width: i64, height: i64): Vec<i64> throws string

The outpaint margins of a width x height picture in pixels (top, bottom, left, right): percents, "T B L R" of the height and width ("": 10 each, at least one above 0); ratio, WIDTH:HEIGHT ("": none), widens the two smaller margins to it. Each is a multiple of 16, at least 16 where it is above 0.

Effects: fail.

`imageOutputOf`

gizlang

function imageOutputOf(j: JsonValue): ImageOutput throws string

The output of a request's outputFormat (or OpenAI's output_format): webp (the default, a tenth of a PNG's bytes), png or jpeg, at outputQuality (or output_compression; 90).

Effects: fail.

`serveImages`

gizlang

function serveImages(sys: System, name: string, place: Hosting.Place, load: (gpu: Gpu) => Result<ImageModel, string>, capacity: i64 = 4): i64 throws string

Serves image model name as giz's inference worker at place: operation image.generate, capacity jobs held; load makes the model on the host's GPU. It returns when the worker stops.

Effects: fail, read, write, console, io.

Methods

`ImageOutput.encode`

gizlang

function ImageOutput.encode(self: ImageOutput, image: Image.Image): bytes

image in the output's format.

Effects: fail.

`ImageModel.caption`

gizlang

function ImageModel.caption(self: ImageModel, sys: System, r: ImageRequest): Result<string, string>

The fuller prompt that the upsampler writes for r. It loads beside the model on the model's Gpu, so the model's pool of dropped buffers goes back first, and its own buffers go back after; it needs what it took at its last use (Gpu.peak), or before that its files' bytes, and where that does not fit this is NO_ROOM.

Effects: fail, read, write, console, io.

`ImageModel.probe`

gizlang

function ImageModel.probe(self: ImageModel, sys: System, side: i64): Result<Vec<i64>, string>

Of one step and the decode of a side × side text-to-image job: the rows, the device bytes (Gpu.peak) of the step, the device bytes of the decode, and the image's tokens.

Effects: fail, read, write, console, io.

`ImageModel.takes`

gizlang

function ImageModel.takes(self: ImageModel, r: ImageRequest): bool throws string

Whether this model takes r as its painter settles it.

Effects: fail, read, write, console, io.

`ImageModel.paint`

gizlang

function ImageModel.paint(self: ImageModel, sys: System, r: ImageRequest, i: i64, progress: (step: i64, total: i64) => bool, probe: bool = false, text: string = ""): Painted throws string

Image i of request r (settled) of the prompt text ("": r.prompt). It fails where the request does not fit the GPU beside the model (fits), but for the probe that measures that.

Effects: fail, read, write, console, io.

`ImageModel.stepped`

gizlang

function ImageModel.stepped(self: ImageModel, probe: bool): i64

For a probe, the device peak of the steps just run; the pool is then trimmed and the peak reset, so that the decode's peak counts only what it holds. Else 0.

Effects: fail, read, write, io.

`ImageModel.measure`

gizlang

function ImageModel.measure(self: ImageModel, sys: System): ImageNeed throws string

What a request needs on this GPU, measured: a 512² and a 1024² job; the bytes of a step for each row between them, and the bytes of the 1024² decode for each token.

Effects: fail, read, write, console, io.

`ImageModel.fits`

gizlang

function ImageModel.fits(self: ImageModel, rows: i64, width: i64, height: i64, pictures: i64): Result<bool, string>

Whether a job of rows (a width × height image with pictures) fits the GPU beside the model now: the more of its steps' need and its decode's (ImageNeed; Gpu.fits). A job that does not would stop the program in Gpu.bytes; it is refused with what to change instead.

Effects: fail, read, write, io.

`ImageModel.make`

gizlang

function ImageModel.make(self: ImageModel, sys: System, r0: ImageRequest, progress: (stage: string, step: i64, total: i64) => bool): Made throws string

The images of request r (checkImage): count of them in turn, from the seeds seed, seed + 1 and on. progress is told the stage ("encoding_prompt", then "generating"), the step done and the steps in all; when it returns false make stops with CANCELLED. The tunings of the kernels it made are kept for the next start.

Effects: fail, read, write, console, io.

Constants

`CANCELLED`

gizlang

const CANCELLED = "cancelled"

The error make gives when its progress callback says to stop.

`NO_ROOM`

gizlang

const NO_ROOM = "no room for the prompt upsampler beside the model on this GPU"

The error make gives when the upsampler does not fit beside the model (the request can go without it).