# gizml:diffusion video and image generation models (diffusion transformers), fast on any hardware. A model family is a recipe: its config.json read as an Arch, and the names of its tensors. Every family runs on the same operations: the linears are gizml:tensor products on the matrix cores (BF16, or INT8 from a W4 checkpoint as production quantizes H3), and the row operations and attention are the kernels below. A GPU runs them; the host runs the same block in f32, which is the reference of the tests. MiniMax H3 (the first family) is one single-stream transformer over a packed sequence of text, conditioning, audio and video rows. Each row has rotary (t, h, w) coordinates and a modulation row: the (timestep, modality) pair whose shift, scale and gate its block applies. Attention is full and unmasked over the whole sequence, so the order of the rows does not matter beyond the layout the caller keeps. ```gizlang import * as Diffusion from "gizml:diffusion"; ``` ## Types ### `Arch` ```gizlang type Arch = { family: string; hidden: i64; layers: i64; refiner: i64; heads: i64; head: i64; ffn: i64; latents: i64; audio: i64; patch: Vec; textDim: i64; freqDim: i64; timeHidden: i64; timeDim: i64; eps: f64; qkEps: f64; finalEps: f64; } ``` What a model computes, as its family's recipe reads config.json. - `family`: `string` - `hidden`: `i64` - `layers`: `i64` - `refiner`: `i64` - `heads`: `i64` - `head`: `i64` - `ffn`: `i64` - `latents`: `i64` - `audio`: `i64` - `patch`: `Vec` - `textDim`: `i64` - `freqDim`: `i64` - `timeHidden`: `i64` - `timeDim`: `i64` - `eps`: `f64` - `qkEps`: `f64` - `finalEps`: `f64` ### `Block` ```gizlang type Block = { w: Vec; bias: Vec; codes: Vec; lora: Vec; } ``` One block's weights, by the indexes below: its 4 linears [out, in] as stored, then its 4 norms (f32). On a GPU a linear is BF16, or W4 when `codes` holds the decoding of each (`w` then holds its codes); on the host it is f32 values (W4: the code bytes as numbers). `bias` holds the linears' biases (f32, one per output) or none; `lora` the A ([rank, in]) and B ([out, rank]) of each linear's LoRA, applied beside it (BF16 on a GPU; a W4 block's A there is INT8 as N.int8Rows makes it, its row scales in B's columns), or none. UP's rows are the gate's and the up projection's in pairs (gluRows; SwiGLU: silu(gate) times up). - `w`: `Vec` - `bias`: `Vec` - `codes`: `Vec` - `lora`: `Vec` ### `Sequence` ```gizlang type Sequence = { rows: i64; map: Tensor; cos: Tensor; sin: Tensor; exact: bool; group: i64; runs: Vec; queries: i64; } ``` The rows of one packed sequence: per row its modulation row (`map`, as f32 values), and the cosines and sines of its rotary angles (per axis its frequencies: H3's t, h and w; FLUX.3's four). Tensors where the block runs. `exact`: its attention in BF16 where 8-bit matrix cores would take it (the reference of N.attentionRows quantized). - `rows`: `i64` - `map`: `M32.Tensor` - `cos`: `M32.Tensor` - `sin`: `M32.Tensor` - `exact`: `bool` - `group`: `i64` - `runs`: `Vec` - `queries`: `i64` ### `Modulation` ```gizlang type Modulation = { attn: Vec; mlp: Vec; } ``` A block's modulation: shift, scale and gate of the attention, then of the MLP, each [modulation rows, hidden] f32. - `attn`: `Vec` - `mlp`: `Vec` ### `Model` ```gizlang type Model = { arch: Arch; gpu: Option; grouped: bool; named: Map; lora: Map; curves: Tensor; map: Tensor; exact: bool; inverse: Vec; head: Vec; refiner: Vec; blocks: Vec; staged: Vec; slots: Vec; kept: Vec; } ``` A model ready to run: its Arch, where it runs, whether its blocks' linears are W4 (production's checkpoint) or BF16, the checkpoint's tensors and its LoRA's, the AdaLN curves of a pruned checkpoint (its table of time embeddings, [grid, rank] f32 on the host; none: the time embedder) and the LoRA's map to them (Model.adaln), the rotary inverse frequencies, the tensors outside the blocks, the token refiner, and the blocks that stay on the GPU (all of them when they fit; the others are `staged` in page-locked host memory and copy to the two `slots` ahead of their use). `exact`: attention in BF16 (Sequence). `kept`: the tables of the schedules it prepared (Model.prepare). - `arch`: `M20.Arch` - `gpu`: `_gizlang_sdk_core.Option` - `grouped`: `bool` - `named`: `_gizlang_sdk_collections.Map` - `lora`: `_gizlang_sdk_collections.Map` - `curves`: `M32.Tensor` - `map`: `M32.Tensor` - `exact`: `bool` - `inverse`: `Vec` - `head`: `Vec` - `refiner`: `Vec` - `blocks`: `Vec` - `staged`: `Vec` - `slots`: `Vec` - `kept`: `Vec` ### `Staged` ```gizlang type Staged = { host: Pinned; bytes: i64; parts: Vec; counts: Vec; } ``` A block in page-locked host memory: the bytes of its GPU tensors (blockParts) one after another, each from a multiple of 256, copied to a slot on the GPU ahead of each use (Gpu.copyAhead). `counts`: Block's lists' lengths. - `host`: `M10.Pinned` - `bytes`: `i64` - `parts`: `Vec` - `counts`: `Vec` ### `Part` ```gizlang type Part = { shape: Vec; dtype: i64; at: i64; size: i64; host: Tensor; } ``` A tensor of a staged block: its shape, type, and bytes from `at`; a host tensor (`at` -1) stays as `host`. - `shape`: `Vec` - `dtype`: `i64` - `at`: `i64` - `size`: `i64` - `host`: `M32.Tensor` ### `Tables` ```gizlang type Tables = { times: Vec; blocks: Vec; final: Vec; } ``` A schedule's modulation, where the model runs: for each block, the shift, scale and gate of its attention and MLP for each row 3 i + modality of the timesteps `times`; and the final layer's shift and scale for each timestep. - `times`: `Vec` - `blocks`: `Vec` - `final`: `Vec` ### `Layout` ```gizlang type Layout = { text: i64; cond: i64; condAudio: i64; audio: i64; kept: i64; video: i64; positions: Vec; } ``` The rows of one request in packed order: text, keyframe video (conditioning), audio (the 2 channels of the audio frames, one channel after the other), then video (latent frames of a grid of patches); and the rotary (t, h, w) of each row. - `text`: `i64` - `cond`: `i64` - `condAudio`: `i64` - `audio`: `i64` - `kept`: `i64` - `video`: `i64` - `positions`: `Vec` ### `Keyframe` ```gizlang type Keyframe = { at: f64; frames: i64; } ``` A keyframe: `frames` latent frames of the target's grid at pixel frame `at` of the target video. - `at`: `f64` - `frames`: `i64` ### `Conditions` ```gizlang type Conditions = { history: i64; audioHistory: i64; audioFirst: i64; kept: i64; } ``` What a request holds beside its keyframes (WanGP's continuation of a video and soundtrack): `history` latent frames of video before the target (the first condition rows), `audioHistory` latent audio frames before it and `audioFirst` at its start (the audio condition rows), and the target's first `kept` latent audio frames given (a soundtrack), which sampling keeps. - `history`: `i64` - `audioHistory`: `i64` - `audioFirst`: `i64` - `kept`: `i64` ### `RefRows` ```gizlang type RefRows = { kind: string; frames: i64; lh: i64; lw: i64; audio: i64; } ``` A reference of ref2va, in the order WanGP lists them (images, videos, audio): its `kind` ("image", "video" or "audio"), `frames` latent frames of `lh` x `lw` latents (patches of 2 x 2), and `audio` latent audio frames (a video's soundtrack or an audio reference). Its rows come after the keyframes' (video) and the audio conditions (audio). - `kind`: `string` - `frames`: `i64` - `lh`: `i64` - `lw`: `i64` - `audio`: `i64` ### `Request` ```gizlang type Request = { layout: Layout; sequence: Sequence; text: Tensor; cond: Tensor; condAudio: Tensor; tags: Vec; } ``` A request ready to sample, where the model runs: its layout and sequence, its text rows after the token refiner, and its condition rows after the patch projections ([rows, hidden] f32): video, then audio. - `layout`: `M20.Layout` - `sequence`: `M20.Sequence` - `text`: `M32.Tensor` - `cond`: `M32.Tensor` - `condAudio`: `M32.Tensor` - `tags`: `Vec` ### `Conditioning` ```gizlang type Conditioning = { text: Tensor; tags: Vec; } ``` The text encoder's rows of a prompt: [rows, textDim] f32 on the host, and each row's tag (see Model.request). - `text`: `M32.Tensor` - `tags`: `Vec` ### `StepCache` ```gizlang type StepCache = { limit: f64; start: i64; first: Vec; rest: Vec; pending: Vec; step: i64; skipped: i64; } ``` A sampler's step cache (First Block Cache, as ParaAttention's), which Flux.sample and Model.sample share: where the first block's residual moved less than `limit` (its mean absolute change over its mean magnitude) since the last full step, a step adds that step's residual of the other blocks instead of running them. 0 runs every block; above 0 the output changes. A sampler calls cacheStart before its blocks, cacheSkip after the first and cacheEnd after the last. Steps before `start` run every block (WanGP's start step). `first` and `rest` hold the last full step's residuals, `pending` this step's first one until cacheEnd; `step` counts the steps, `skipped` those that reused them. - `limit`: `f64` - `start`: `i64` - `first`: `Vec` - `rest`: `Vec` - `pending`: `Vec` - `step`: `i64` - `skipped`: `i64` ### `Decoder` ```gizlang type Decoder = { arch: Arch; gpu: Option; blocks: Vec; gates: Vec; head: Vec; mean: Vec; std: Vec; inverse: Vec; } ``` The decoder of H3's video VAE (AutoencoderKLMiniMaxH3): each latent voxel of a clip is a token of a ViT (heads of 64, biases, a gate a channel for each branch, 4 register tokens and a zero token after the voxels, rotary angles of positions in [-1, 1) times 2 pi), and becomes 4 x 16 x 16 pixels. Videos decode in clips of 7 latent frames, 5 apart, and in tiles of 256 x 256 pixels that overlap by 64 at least, blended as the reference blends them. - `arch`: `M20.Arch` - `gpu`: `_gizlang_sdk_core.Option` - `blocks`: `Vec` - `gates`: `Vec` - `head`: `Vec` - `mean`: `Vec` - `std`: `Vec` - `inverse`: `Vec` ### `Encoder` ```gizlang type Encoder = { gpu: Option; named: Map; zeros: bool; mean: Vec; std: Vec; taps: Map; } ``` A VAE encoder (loadEncoder): its tensors by name (a 3 x 3 convolution as [out, in 9], BF16 on a GPU), where it runs, its padding (`zeros`: BFL's; else reflected: H3's), and the latents' mean and deviation by channel. - `gpu`: `_gizlang_sdk_core.Option` - `named`: `_gizlang_sdk_collections.Map` - `zeros`: `bool` - `mean`: `Vec` - `std`: `Vec` - `taps`: `_gizlang_sdk_collections.Map` ### `AudioConv` ```gizlang type AudioConv = { w: Tensor; bias: Vec; } ``` A 1-D convolution: its weight ([out, in, k], or [in, out, k] transposed) and its bias, or none. - `w`: `M32.Tensor` - `bias`: `Vec` ### `AudioSnake` ```gizlang type AudioSnake = { alpha: Tensor; inverse: Tensor; filters: Tensor; } ``` BigVGAN's alias-free SnakeBeta of `channels` channels: per channel e^alpha and 1 / (e^beta + 1e-9), and the 12 taps of the 2x upsampler then of the downsampler (filters [24] f32). - `alpha`: `M32.Tensor` - `inverse`: `M32.Tensor` - `filters`: `M32.Tensor` ### `AudioBlock` ```gizlang type AudioBlock = { size: i64; first: Vec; second: Vec; snakes: Vec; } ``` One anti-aliased multi-periodicity block (AMPBlock1): for each dilation a dilated convolution and a plain one, each after its own Snake. - `size`: `i64` - `first`: `Vec` - `second`: `Vec` - `snakes`: `Vec` ### `BigVgan` ```gizlang type BigVgan = { gpu: Option; pre: AudioConv; ups: Vec; rates: Vec; blocks: Vec; dilations: Vec; post: AudioSnake; out: AudioConv; } ``` BigVGAN v2's generator (AMPBlock1, SnakeBeta): conv_pre (7 taps), then each stage a transposed convolution that upsamples by its rate and the mean of its residual blocks (kernels 3, 7, 11 by default, dilations 1, 3, 5), then the last Snake and conv_post (7 taps). H3's audio VAE decodes with one; LTX-2.5's vocoder and its bandwidth extension are two more. - `gpu`: `_gizlang_sdk_core.Option` - `pre`: `M20.AudioConv` - `ups`: `Vec` - `rates`: `Vec` - `blocks`: `Vec` - `dilations`: `Vec` - `post`: `M20.AudioSnake` - `out`: `M20.AudioConv` ### `AudioDecoder` ```gizlang type AudioDecoder = { stats: Vec; proj: AudioConv; vgan: BigVgan; } ``` H3's audio VAE decoder (AutoencoderKLMiniMaxH3Audio): latents of 32 channels to mono 32 kHz waveforms, one for each stereo channel. `stats` (the mean and deviation, [1, 32]) undo the latents' normalization (WanGP's MiniMaxH3AudioVAE), `proj` is the input projection, `vgan` the generator. - `stats`: `Vec` - `proj`: `M20.AudioConv` - `vgan`: `M20.BigVgan` ### `AudioEncoder` ```gizlang type AudioEncoder = { gpu: Option; convs: Map; named: Map; } ``` H3's audio VAE encoder (AutoencoderKLMiniMaxH3Audio.encode, WanGP's MiniMaxH3AudioVAE): a mono 32 kHz waveform to latents of 32 channels, one frame a AUDIO_HOP samples. DAC's encoder: a convolution of 7 taps to 64 channels, then stages that double the channels (three residual units of Snake, a dilated convolution of 7 taps, Snake and a 1 x 1 convolution; Snake; a convolution of 2 stride taps at stride) to 2048, Snake and a convolution of 3 taps. Then the pre-block: a causal attention of 8 heads whose heads are averaged and pooled to 32 values, beside a projection of the layer-normed rows to 32; then a GeGLU MLP; then the mean projection, normalized as decode undoes it. The weights (weight norm applied) stay f32 on the host and go to `gpu` for each encode. - `gpu`: `_gizlang_sdk_core.Option` - `convs`: `_gizlang_sdk_collections.Map` - `named`: `_gizlang_sdk_collections.Map` ### `Flux` ```gizlang type Flux = { arch: Arch; gpu: Option; time: Vec; target: Vec; early: Vec; single: Vec; imageIn: Tensor; referenceIn: Tensor; textIn: Tensor; finalMod: Tensor; finalOut: Tensor; textBlocks: Vec; imageBlocks: Vec; referenceBlocks: Vec; blocks: Vec; } ``` A FLUX.3 model where it runs: the time and target-time embedders (in, out), the modulation weights of the early and the joint streams (by mode), the input projections (image, reference images, text), the final layer's modulation and projection, and the blocks (text, image, reference, joint), their linears as Block.quantized makes them on a GPU. - `arch`: `M20.Arch` - `gpu`: `_gizlang_sdk_core.Option` - `time`: `Vec` - `target`: `Vec` - `early`: `Vec` - `single`: `Vec` - `imageIn`: `M32.Tensor` - `referenceIn`: `M32.Tensor` - `textIn`: `M32.Tensor` - `finalMod`: `M32.Tensor` - `finalOut`: `M32.Tensor` - `textBlocks`: `Vec` - `imageBlocks`: `Vec` - `referenceBlocks`: `Vec` - `blocks`: `Vec` ### `RefSlot` ```gizlang type RefSlot = { first: i64; count: i64; rows: Tensor; record: bool; } ``` A joint block's reference rows kept across steps (Flux.sample's `refresh`): their keys and values, [count, 2 width] BF16 in `rows`. A step that records copies them from rows `first` to `first` + `count` of the block's qkv; the others, which update only the rows before `first`, put them back there. - `first`: `i64` - `count`: `i64` - `rows`: `M32.Tensor` - `record`: `bool` ### `FluxReference` ```gizlang type FluxReference = { tokens: Tensor; height: i64; width: i64; } ``` A reference image of a FLUX.3 request (fluxReference): its latent tokens ([height × width, 128] f32, Encoder.fluxTokens) on a grid of height by width. - `tokens`: `M32.Tensor` - `height`: `i64` - `width`: `i64` ### `FluxRequest` ```gizlang type FluxRequest = { text: Tensor; references: Vec; grid: Vec; sequences: Vec; } ``` A FLUX.3 request where the model runs: the text's rows ([text, hidden] f32, through txt_in), the reference images' rows (one [references, hidden] tensor through emb_in.image_cond, or none), the image's token grid (latent height / 2 by width / 2) and the sequences of its rows, by FLUX_IMAGE_ROWS and FLUX_TEXT_ROWS (the early blocks of those streams), FLUX_JOINT ([image | text | references]: the rows that change in a step come first) and, with references, FLUX_REFERENCE_ROWS (their rows alone) and FLUX_ACTIVE (the joint one updating the image and the text only, as the steps of Flux.sample's `refresh` run it). A record of the sequences themselves would exceed the compiler's limit. - `text`: `M32.Tensor` - `references`: `Vec` - `grid`: `Vec` - `sequences`: `Vec` ### `ImageDecoder` ```gizlang type ImageDecoder = { gpu: Option; named: Map; channels: i64; mean: Tensor; std: Tensor; } ``` BFL's image VAE decoder (loadImageDecoder): its tensors by name, where it runs, its latent channels, and the latent tokens' mean and std. - `gpu`: `_gizlang_sdk_core.Option` - `named`: `_gizlang_sdk_collections.Map` - `channels`: `i64` - `mean`: `M32.Tensor` - `std`: `M32.Tensor` ### `TextEncoder` ```gizlang type TextEncoder = { engine: L.Engine; } ``` FLUX.3's text encoder (loadTextEncoder): Qwen3-VL-4B's first layers on a gizml:llm engine. - `engine`: `M23.Engine` ### `Upsampler` ```gizlang type Upsampler = { engine: L.Engine; vision: L.Vision; prompts: JsonValue; } ``` FLUX.3's prompt upsampler (loadUpsampler): Qwen3.5's language model, its vision tower and the system prompts. - `engine`: `M23.Engine` - `vision`: `M23.Vision` - `prompts`: `gizlang_json.Json` ### `FluxBox` ```gizlang type FluxBox = { bbox: Vec; desc: string; } ``` A box of a layout prompt: [y_min, x_min, y_max, x_max] on a 0-1000 canvas ((0, 0) at the top left) and what belongs in it. - `bbox`: `Vec` - `desc`: `string` ### `Klein` ```gizlang type Klein = { gpu: Gpu; flux: Ref; text: Ref; encoder: Ref; decoder: Ref; } ``` FLUX.3 klein ready to paint (loadKlein): its parts, each by reference (together they are more than a record holds) and already on `gpu`. - `gpu`: `M10.Gpu` - `flux`: `Ref` - `text`: `Ref` - `encoder`: `Ref` - `decoder`: `Ref` ### `KleinJob` ```gizlang type KleinJob = { request: FluxRequest; width: i64; height: i64; } ``` What `condition` makes for `sample` and `decode`: the request and the image's size in pixels. - `request`: `M20.FluxRequest` - `width`: `i64` - `height`: `i64` ### `H3Text` ```gizlang type H3Text = { engine: L.Engine; vision: L.Vision; } ``` H3's text encoder (loadH3Text): the language model and Qwen3-VL's vision tower. - `engine`: `M23.Engine` - `vision`: `M23.Vision` ### `Presented` ```gizlang type Presented = { kind: string; picture: Tensor; times: Vec; } ``` An item H3 shows before the prompt: "image" (`picture` [3, h, w] f32 in [0, 1] on the host, h and w multiples of 32), "video" (`picture` [n, 3, h, w], and `times`: each frame's second) or "audio" (its marker alone). - `kind`: `string` - `picture`: `M32.Tensor` - `times`: `Vec` ### `ZImage` ```gizlang type ZImage = { arch: Arch; gpu: Option; time: Vec; capNorm: Tensor; capIn: Vec; imageIn: Vec; pads: Vec; mods: Vec>; finalMod: Vec; finalOut: Vec; context: Vec; noise: Vec; blocks: Vec; } ``` Z-Image where it runs: the time embedder's two linears and biases (on the host), the caption embedder's norm, linear and bias, the image embedder's linear and bias, the prompt's and the image's padding rows, each modulated block's modulation linear and bias (the noise refiners, then the blocks), the final layer's modulation and output (linear, bias each), and the blocks: context refiners, noise refiners and the 30, their linears as Block.quantized makes them on a GPU. - `arch`: `M20.Arch` - `gpu`: `_gizlang_sdk_core.Option` - `time`: `Vec` - `capNorm`: `M32.Tensor` - `capIn`: `Vec` - `imageIn`: `Vec` - `pads`: `Vec` - `mods`: `Vec>` - `finalMod`: `Vec` - `finalOut`: `Vec` - `context`: `Vec` - `noise`: `Vec` - `blocks`: `Vec` ### `ZImageRequest` ```gizlang type ZImageRequest = { prompt: Tensor; height: i64; width: i64; image: Sequence; joint: Sequence; } ``` What ZImage.request makes for the steps: the prompt's rows after the context refiners ([prompt rows padded, hidden] f32), the image's size in tokens, and the sequences of the image's rows (padded) and of [prompt; image]. - `prompt`: `M32.Tensor` - `height`: `i64` - `width`: `i64` - `image`: `M20.Sequence` - `joint`: `M20.Sequence` ### `ZImageText` ```gizlang type ZImageText = { engine: L.Engine; } ``` Z-Image's text encoder (loadZImageText): Qwen3-4B's first ZIMAGE_TEXT_LAYERS layers on a gizml:llm engine. The prompt in Qwen's chat template, with no padding, gives the residual stream after them (no final norm), as ComfyUI's Qwen3_4BModel (layer_idx -2) does. - `engine`: `M23.Engine` ### `ZImagePipeline` ```gizlang type ZImagePipeline = { gpu: Gpu; model: Ref; text: Ref; decoder: Ref; } ``` Z-Image Turbo ready to paint (loadZImagePipeline): the model, its text encoder and FLUX.1's VAE decoder, each by reference and on `gpu`. As Klein, in stages a caller can time: `condition`, `sample` and `decode`. - `gpu`: `M10.Gpu` - `model`: `Ref` - `text`: `Ref` - `decoder`: `Ref` ### `H3` ```gizlang type H3 = { model: Model; decoder: Decoder; encoder: Encoder; audio: AudioDecoder; listen: Vec; text: Vec; textHost: string; textPort: i64; folder: string; spare: i64; } ``` H3 loaded for a service: the transformer and both VAEs on one GPU, and the text encoder: in this process (`text`, from the folder's text/ on the GPU) or giz's shared H3 text encoder service at textHost:textPort. - `model`: `M20.Model` - `decoder`: `M20.Decoder` - `encoder`: `M20.Encoder` - `audio`: `M20.AudioDecoder` - `listen`: `Vec` - `text`: `Vec` - `textHost`: `string` - `textPort`: `i64` - `folder`: `string` - `spare`: `i64` ### `H3Job` ```gizlang type H3Job = { prompt: string; width: i64; height: i64; frames: i64; fps: i64; steps: i64; shift: f64; seed: i64; cache: f64; cacheStart: i64; start: Vec; end: Vec; injected: Vec; positions: Vec; history: Vec; sound: Vec; soundtrack: bool; control: Vec; mask: Vec; denoising: f64; masking: f64; solver: string; frozen: bool; refs: Vec; relative: f64; } ``` One request to H3, as giz's worker passes it on: the prompt; the canvas width x height (multiples of 32) and the frames (rounded up to 5 + 17 k) at `fps`; `steps` Euler passes at `shift`; the noise's `seed`; the step cache's limit and first step (stepCache; 0 runs every block); the first frame `start` and last frame `end` (none or one each) and the `injected` frames at `positions` (0-based frames; -1 the last). A first frame sets the video's size: the canvas's area at the picture's aspect, each side a multiple of 32 (WanGP's fit_canvas 0). `history` (none or one [3, n, height, width] in [-1, 1] on the host) continues a video: its frames come first, the last as the first frame, the others encoded (a sliding window's overlap, n = 1 + 17 k). `sound` (none or one [2, samples] at AUDIO_SAMPLE_RATE) is the audio from the history's start: the history's part continues the audio and, with `soundtrack`, the rest is kept as the target's (WanGP's audio prompt "A" or "K"). `control` (the frames of a control video from the window's first frame, each width * height * 3 RGB bytes) and `mask` (a byte of 0 or 1 a pixel of each frame of it, 1 for the area to make; none for no mask) edit that video (WanGP's video to video): it is encoded, and from the pass `denoising` (0 to 1) starts its latents are put back at each pass in the noise of that pass, but for the mask's area during the `masking` share of the passes. - `prompt`: `string` - `width`: `i64` - `height`: `i64` - `frames`: `i64` - `fps`: `i64` - `steps`: `i64` - `shift`: `f64` - `seed`: `i64` - `cache`: `f64` - `cacheStart`: `i64` - `start`: `Vec` - `end`: `Vec` - `injected`: `Vec` - `positions`: `Vec` - `history`: `Vec` - `sound`: `Vec` - `soundtrack`: `bool` - `control`: `Vec` - `mask`: `Vec` - `denoising`: `f64` - `masking`: `f64` - `solver`: `string` - `frozen`: `bool` - `refs`: `Vec` - `relative`: `f64` ### `H3Ref` ```gizlang type H3Ref = { kind: string; image: Vec; frames: Vec; width: i64; height: i64; sound: Vec; } ``` A reference of ref2va: an "image" (`image`, one; shown at `relative` per cent of the video's area), a "video" (its RGB `frames`, 17 k + 5 of them at `width` x `height`, multiples of 32; `sound`, its soundtrack when it has one) or an "audio" (`sound`, one wave [2, samples] at AUDIO_SAMPLE_RATE). - `kind`: `string` - `image`: `Vec` - `frames`: `Vec` - `width`: `i64` - `height`: `i64` - `sound`: `Vec` ### `H3Output` ```gizlang type H3Output = { video: Tensor; audio: Tensor; } ``` A job's video ([3, frames, height, width] RGB in [0, 1] on the GPU) and its stereo audio ([2, samples] at AUDIO_SAMPLE_RATE on the host). - `video`: `M32.Tensor` - `audio`: `M32.Tensor` ## Functions ### `archOf` ```gizlang function archOf(config: JsonValue): Arch throws string ``` The Arch of a model by its transformer's config.json; an error names what this module does not run yet. Effects: fail. ### `loadBlock` ```gizlang function loadBlock(sys: System, named: Map, lora: Map, prefix: string, gpu: Option, interleaved: bool): Block throws string ``` Block `prefix` (as "blocks.3.") of the checkpoint's tensors on `gpu` when there is one, with the LoRA `lora` (an empty map: none): W4 or ComfyUI's INT8 ConvRot as stored when the checkpoint holds it so (w4Block, int8Block), else BF16 with the LoRA merged, its QKV rows a head at a time when `interleaved` (see weightAt; production's files hold them grouped). ComfyUI's scaled FP8 is quantized to INT8 (Block.quantized) after the merge: as BF16 it would take twice the GPU memory. Effects: fail, read, write, console, io. ### `sequence` ```gizlang function sequence(positions: Vec, map: Vec, inverse: Vec, gpu: Option, exact: bool = false, group: i64 = 0): Sequence ``` The sequence of `positions` (rows of a position per axis: H3's t, h, w) whose rows use modulation rows `map`, with the rotary inverse frequencies `inverse` of each axis (H3's rope.inv_freq: head / 8 of them), on `gpu` when there is one. Rows attend within groups of `group` rows (0: all of them), as tiles of a VAE. Effects: fail, read, write, console, io. ### `sequenceOf` ```gizlang function sequenceOf(positions: Vec, map: Vec, inverses: Vec>, gpu: Option, exact: bool = false, group: i64 = 0): Sequence ``` `sequence` with each axis its own inverse frequencies (Z-Image's 16, 24 and 24). Effects: fail, read, write, console, io. ### `load` ```gizlang function load(sys: System, path: string, lora: string, gpu: Option, spare: i64, map: string = ""): Model throws string ``` The model at `path`: a transformer folder (config.json and its safetensors), or one .safetensors file of H3 (as production's W4 checkpoint), with the LoRA file `lora` unless it is "" (merged into BF16 weights, beside W4 ones), on `gpu` when there is one. Blocks stay on the GPU while its free memory keeps `spare` bytes for a pass (Model.fill stages the rest). A pruned checkpoint (AdaLN curves) takes a LoRA of the full AdaLN through `map` (WanGP's lora_affine_maps file of the model's table). Effects: fail, read, write, console, io. ### `stagedBytes` ```gizlang function stagedBytes(b: Block): i64 ``` The bytes of b's staged copy (Staged.bytes) without making it. Effects: fail. ### `stage` ```gizlang function stage(g: Gpu, b: Block): Staged ``` Block b in page-locked host memory (Staged): its GPU tensors copied, its host ones kept. Effects: fail, read, write, io. ### `sinusoids` ```gizlang function sinusoids(times: Vec, dim: i64): Tensor ``` Rows [cos | sin] of each time times the dim / 2 frequencies 10000^(-i / (dim / 2)), in f32 as diffusers' Timesteps (flip_sin_to_cos) computes them: the timestep embedding of H3 and FLUX.3. Effects: fail. ### `noConditions` ```gizlang function noConditions(): Conditions ``` Conditions with no history and no kept audio. ### `layout` ```gizlang function layout(text: i64, frames: i64, height: i64, width: i64, audioFrames: i64, keyframes: Vec, more: Conditions, refs: Vec = []): Layout ``` The rows of a request: `text` rows of text, the video's `frames` latent frames of `height` x `width` latents (patches of 2 x 2), `audioFrames` latent audio frames, and the keyframes and conditions. Time advances 1 a text row and a latent audio frame, and 5/3 a pixel frame; patches are on a grid of 32 around 16, scaled by the frame's aspect. The history comes first in time (from the text's end); the target, its keyframes and its audio after it. Effects: fail. ### `sharedText` ```gizlang function sharedText(sys: System, host: string, port: i64, prompt: string, items: Vec = []): Conditioning throws string ``` The conditioning of `prompt` after `items` (Presented) from a shared H3 text encoder at host:port: giz's wgp-api with H3_TEXT_ENCODER_SERVER (POST /api/h3/text-embeddings; it answers with the safetensors of "context" (1, rows, textDim) and "tags" (rows)). The request's bytes are those of its other clients, so its cache serves both. Effects: fail, read, write, console, io. ### `scheduleTimes` ```gizlang function scheduleTimes(steps: i64, shift: f64): Vec ``` The timesteps of the passes of a schedule of `steps` at `shift` (Model.sample's): each pass's video, audio and keyframe timesteps, each once, then AUDIO_CONDITION_TIME. Effects: fail. ### `sigmas` ```gizlang function sigmas(steps: i64, shift: f64): Vec ``` The sigmas of H3's flow schedule for `steps` passes: linspace(1, 0, steps + 1) shifted by `shift` (s σ / (1 + (s - 1) σ)), in f32 as the reference, without repeats. Effects: fail. ### `resCoefficients` ```gizlang function resCoefficients(values: Vec): Vec> ``` WanGP's _res_multistep_coefficients: for each pass of the schedule `values`, the weights of the sample, of the denoised sample and of the pass before's (second order RES, arXiv:2308.02157); a first pass, and the last (to sigma 0), are Euler's. Effects: fail. ### `stepCache` ```gizlang function stepCache(limit: f64, start: i64 = 0): StepCache ``` A step cache with LIMIT, which runs every block before step START. ### `patchify` ```gizlang function patchify(z: Tensor): Tensor ``` The video rows ([frames h w, latents x 4], f32 on the host) of latents [latents, frames, 2 h, 2 w] (unpatchify's inverse). Effects: fail, read, write, console, io. ### `unpatchify` ```gizlang function unpatchify(rows: Tensor, latents: i64, frames: i64, h: i64, w: i64): Tensor ``` The video latents ([latents, frames, 2 h, 2 w], f32 on the host) of video rows [frames h w, latents x 4] (a patch of 2 x 2 latents a row, channel-major: column c 4 + 2 p + q holds channel c at (2 y + p, 2 x + q)). Effects: fail, read, write, console, io. ### `loadDecoder` ```gizlang function loadDecoder(sys: System, folder: string, gpu: Option, quantized: bool = true): Decoder throws string ``` The decoder of a VAE folder (config.json and its safetensors, diffusers' names) or of WanGP's one .safetensors file (its names and H3's config; minimax_h3_video_vae_fp8mix holds E4M3 linears) on `gpu` when there is one, its linears BF16, or INT8 through ConvRot (Block.quantized) when `quantized` and the GPU has INT8 matrix cores and the widths allow it. Effects: fail, read, write, console, io. ### `loadEncoder` ```gizlang function loadEncoder(sys: System, path: string, gpu: Option): Encoder throws string ``` The encoder of a VAE file on `gpu` when there is one: WanGP's H3 VAE (minimax_h3_video_vae_fp8mix: its encoder is BF16, its convolutions 3D) or BFL's FLUX.2 VAE (flux2-vae, BFL's names; its batch norm's statistics as the mean and deviation). Effects: fail, read, write, console, io. ### `loadBigVgan` ```gizlang function loadBigVgan(sys: System, named: Map, prefix: string, rates: Vec, kernels: Vec, dilations: Vec, numbered: bool, gpu: Option): BigVgan throws string ``` BigVGAN's generator under `prefix` of `named` (weight-normed or plain convolutions), its stages upsampling by `rates`, on `gpu` when there is one. H3's names (`numbered`) put each stage's transposed convolution under ups.i.0 and a block's Snakes under activations.0 to 5 in turn; LTX's under ups.i, and acts1.e and acts2.e. Effects: fail, read, write, console, io. ### `loadAudioDecoder` ```gizlang function loadAudioDecoder(sys: System, path: string, gpu: Option): AudioDecoder throws string ``` The decoder of H3's audio VAE file (MiniMax-H3-audio_vae_fp32.safetensors), f32, on `gpu` when there is one. Effects: fail, read, write, console, io. ### `loadAudioEncoder` ```gizlang function loadAudioEncoder(sys: System, path: string, gpu: Option): AudioEncoder throws string ``` The encoder of H3's audio VAE file (MiniMax-H3-audio_vae_fp32.safetensors), to run on `gpu` when there is one. Effects: fail, io. ### `fluxArch` ```gizlang function fluxArch(): Arch ``` FLUX.3 Klein's sizes: BFL's single file holds no config, and these match its tensors (checked as it loads). Effects: fail. ### `comfyInt8Rows` ```gizlang function comfyInt8Rows(sys: System, named: Map, names: Vec, gpu: Gpu): Vec throws string ``` Linears `names` of a ComfyUI INT8 ConvRot file (each name's .weight and .weight_scale: INT8 rows rotated by the regular Hadamard of 256, a scale a row) stacked by rows on `gpu`, as N.int8Rows' products read them (canonicalInto): [the INT8 weights [rows, in], their scales [1, rows] F32]. Effects: fail, read, write, console, io. ### `loadFlux` ```gizlang function loadFlux(sys: System, path: string, gpu: Option, bits: i64 = 8): Flux throws string ``` FLUX.3 Klein from its checkpoint (BFL's single .safetensors, BF16), on `gpu` when there is one: the blocks' linears INT8 (ConvRot), or INT4 with 4 `bits` where the GPU has INT4 matrix cores (K.hasI4Mma; else INT8): faster, at a loss of quality. A GPU without INT8 matrix cores keeps them BF16 (Block.quantized). Effects: fail, read, write, console, io. ### `fluxSigmas` ```gizlang function fluxSigmas(steps: i64, shift: f64): Vec ``` FLUX.3's sigmas for `steps` steps: 1 to 1 / steps evenly, each shifted by `shift` (s σ / (1 + (s - 1) σ)), then 0. Effects: fail. ### `fluxSize` ```gizlang function fluxSize(width: i64, height: i64, megapixels: f64, multiple: i64): Vec ``` The size (width, height) of `megapixels` × 1024² pixels at the aspect of width by height, each a multiple of `multiple`, as ComfyUI's ImageScaleToTotalPixels (an edit workflow's output: 4 megapixels, multiples of 16). Effects: fail. ### `fluxReference` ```gizlang function fluxReference(e: Encoder, p: Img.Image, megapixels: f64 = 1.0): FluxReference ``` Picture `p` as a FLUX.3 reference image, as ComfyUI's ReferenceMultiLatent makes one: at `megapixels` × 1024² (its "resize", 1 by default; Pillow's LANCZOS), cut in the middle to multiples of 16 (VAE encode's crop) and encoded (Encoder.fluxTokens). Klein.condition first scales the first reference to the output's size, as the edit workflows do. Effects: fail, read, write, console, io. ### `loadImageDecoder` ```gizlang function loadImageDecoder(sys: System, path: string, gpu: Option): ImageDecoder throws string ``` The decoder of BFL's VAE file (FLUX.2's or FLUX.1's ae, BFL's names), on `gpu` when there is one. Effects: fail, read, write, console, io. ### `pixelsImage` ```gizlang function pixelsImage(x: Tensor): Img.Image ``` Pixels [3, h, w] in [-1, 1] (ImageDecoder.decode) as an RGB image: each (x + 1) × 127.5, clamped and rounded. Effects: fail, read, write, console, io. ### `fluxNoise` ```gizlang function fluxNoise(seed: u64, height: i64, width: i64): Tensor ``` Seed `seed`'s noise of FLUX latents of `height` × `width` tokens (an image of 16 × as many pixels) as ComfyUI draws it (torch.randn [1, 128, height, width] on the CPU), as tokens [height × width, 128]: a seed gives ComfyUI's image. Effects: fail, read, write, console, io. ### `loadTextEncoder` ```gizlang function loadTextEncoder(sys: System, folder: string, gpu: Option): TextEncoder throws string ``` The text encoder of Qwen3-VL-4B converted by gizml:llm (L.convert with its first 32 layers), on `gpu` (none: the host). Effects: fail, read, write, console, io. ### `parseFluxBoxes` ```gizlang function parseFluxBoxes(text: string): Vec throws string ``` The boxes of a JSON list of {"bbox": [y0, x0, y1, x1], "desc": text} (the rows of ComfyUI's layout node). Effects: fail. ### `convertUpsampler` ```gizlang function convertUpsampler(sys: System, from: string, to: string): i64 throws string ``` Converts BFL's flux-3-image-klein-pup folder to `to`: the language model by gizml:llm (4-bit layers, as ComfyUI's w4a8 file, and an 8-bit head) with its vision tower, and system_prompts.json. Returns the bytes written. Effects: fail, read, write, console, io. ### `loadUpsampler` ```gizlang function loadUpsampler(sys: System, folder: string, gpu: Option): Upsampler throws string ``` The upsampler in `folder` (convertUpsampler's), on `gpu` (none: the host). Effects: fail, read, write, console, io. ### `fluxBoxes` ```gizlang function fluxBoxes(boxes: Vec): string ``` The boxes as Python's json.dumps writes them (ComfyUI's Flux3LayoutPrompt and BFL's bbox.py): rows of id ("obj_1", ...), bbox and desc. Effects: fail. ### `fluxLayout` ```gizlang function fluxLayout(prompt: string, boxes: Vec): string ``` The prompt of a layout without the upsampler: the boxes' table after the prompt (ComfyUI's Flux3LayoutPrompt). Effects: fail. ### `textBits` ```gizlang function textBits(): L.Bits ``` The bits that FLUX.3's text encoder is converted with (convertTextEncoder; gizml:llm's L.converted takes them too). ### `convertTextEncoder` ```gizlang function convertTextEncoder(sys: System, from: string, to: string): i64 throws string ``` Converts Qwen3-VL-4B's Hugging Face folder `from` to the text encoder in folder `to`: its first TEXT_LAYERS layers by gizml:llm (textBits). Returns the bytes written. Effects: fail, read, write, console, io. ### `loadKlein` ```gizlang function loadKlein(sys: System, model: string, vae: string, text: string, gpu: Gpu, bits: i64 = 8): Klein throws string ``` FLUX.3 klein on `gpu`: the model in BFL's single file `model` (loadFlux, with its `bits`), the VAE's in `vae` (BFL's ae, both the decoder and the encoder) and the text encoder in `text` (convertTextEncoder's folder). All parts take their buffers from this one Gpu, so its pool and its measures (Gpu.fits, Gpu.peak) hold all of them. Effects: fail, read, write, console, io. ### `convertH3Text` ```gizlang function convertH3Text(sys: System, from: string, to: string): i64 throws string ``` Converts H3's text encoder (DeepBeepMeep's Qwen3-VL-32B-Instruct folder: config.json, the tokenizer and the layer-50 BF16 safetensors) to folder `to` by gizml:llm (4-bit layers, L.convert), with its vision tower. Returns the bytes written. Effects: fail, read, write, console, io. ### `loadH3Text` ```gizlang function loadH3Text(sys: System, folder: string, gpu: Option, room: i64 = 805_306_368): H3Text throws string ``` H3's text encoder in `folder` (convertH3Text's), on `gpu` (none: the host), its KV pool leaving `room` bytes free for the vision tower's pass over a picture beside the card's reserve (by default H3_TEXT_PICTURE_ROOM). Effects: fail, read, write, console, io. ### `serveH3Text` ```gizlang function serveH3Text(sys: System, listener: Listener, encoder: H3Text): i64 throws string ``` Serves `encoder` on `listener` as giz's shared H3 text encoder (wgp-api's; sharedText is a client): POST /api/h3/text-embeddings takes the form fields prompt, manifest (a JSON list of items: "image" or "video" with the name of its frames, [1, h, w, 3] ([n, h, w, 3] and "timestamps" for a video) in [0, 1], or "audio") and tensors (their safetensors), and answers the safetensors of "context" (1, rows, 5120) BF16 and "tags" (rows) I64; GET /v1/health answers whether it is ready. Requests take turns on the encoder. Effects: fail, read, write, console, io. ### `zImageArch` ```gizlang function zImageArch(): Arch ``` Z-Image's sizes: ComfyUI's file holds no config, and these match its tensors (checked as it loads). Effects: fail. ### `loadZImage` ```gizlang function loadZImage(sys: System, path: string, gpu: Option, bits: i64 = 8): ZImage throws string ``` Z-Image Turbo from ComfyUI's file (z_image_turbo_bf16.safetensors), on `gpu` when there is one: the blocks' linears INT8 (ConvRot), or INT4 with 4 `bits` where the GPU has INT4 matrix cores (K.hasI4Mma); BF16 on a GPU without INT8 matrix cores (Block.quantized). Effects: fail, read, write, console, io. ### `zImageSigmas` ```gizlang function zImageSigmas(steps: i64, shift: f64): Vec ``` Z-Image's sigmas for `steps` steps, as ComfyUI's simple scheduler takes them from ModelSamplingAuraFlow's table of 1000 (multiplier 1): step k at t = (1000 - floor(k 1000 / steps)) / 1000 shifted by `shift`, then 0. Effects: fail. ### `zImageNoise` ```gizlang function zImageNoise(seed: u64, height: i64, width: i64): Tensor ``` Seed `seed`'s noise as Z-Image's tokens ([height × width, 64] f32 on the host): torch.randn of the latent [1, 16, 2 height, 2 width] on the CPU (ComfyUI's prepare_noise), patchified. Effects: fail, read, write, console, io. ### `convertZImageText` ```gizlang function convertZImageText(sys: System, from: string, to: string): i64 throws string ``` Converts Qwen3-4B (a folder of its config.json, tokenizer files and BF16 safetensors, as ComfyUI's qwen_3_4b) to Z-Image's text encoder in folder `to`: its first ZIMAGE_TEXT_LAYERS layers by gizml:llm (textBits). Returns the bytes written. Effects: fail, read, write, console, io. ### `loadZImageText` ```gizlang function loadZImageText(sys: System, folder: string, gpu: Option): ZImageText throws string ``` Z-Image's text encoder in `folder` (convertZImageText's), on `gpu` (none: the host). Effects: fail, read, write, console, io. ### `zImageTokens` ```gizlang function zImageTokens(x: Tensor): Tensor ``` Z-Image's tokens ([height × width, 64], each (y, x, channel)) as ImageDecoder.decode reads them: (channel, y, x). Effects: fail, read, write, console, io. ### `loadZImagePipeline` ```gizlang function loadZImagePipeline(sys: System, model: string, vae: string, text: string, gpu: Gpu, bits: i64 = 8): ZImagePipeline throws string ``` Z-Image Turbo on `gpu` from ComfyUI's model file (loadZImage, its linears of `bits`), the text encoder's folder (convertZImageText) and FLUX.1's ae.safetensors. Effects: fail, read, write, console, io. ### `convertH3` ```gizlang function convertH3(sys: System, sources: Vec, text: string, to: string): i64 throws string ``` Makes the model folder `to` from WanGP's files: `sources` are the transformer, its LoRA, the LoRA's AdaLN map, the video VAE and the audio VAE (h3Files' order), `text` the text encoder converted by convertH3Text ("" for none: the service then uses a shared one). Each file is linked where `to` is on the same file system, else copied. Effects: fail, read, write, console, io. ### `loadH3` ```gizlang function loadH3(sys: System, folder: string, gpu: Gpu, text: string, spare: i64, steps: Vec): Result ``` H3 from model folder `folder` (convertH3) on `gpu`. `text` picks the text encoder: "local" (the folder's, in this process: about 13 GB more on the GPU) or "HOST:PORT" of a shared service (serveH3Text, or production's wgp-api). The tables of the schedules of `steps` (at H3's shift of 12; Model.prepare) are made first; then the transformer's blocks stay on the GPU while `spare` bytes stay free beside them, and the rest copy in at each pass. Effects: fail, read, write, console, io. ### `h3Frames` ```gizlang function h3Frames(frames: i64): i64 ``` The frames WanGP makes for a request of `frames`: at least 5, then up to 5 + 17 k. Effects: fail. ### `latentMask` ```gizlang function latentMask(mask: Vec, from: i64, n: i64, w: i64, h: i64, frames: i64, lh: i64, lw: i64): Vec ``` The mask of the source's `frames` latent frames ([frames, lh, lw] of 0 or 1) from `mask` (a byte of 0 or 1 a pixel of each frame, w x h; `n` frames from frame `from` on, and 1 past the mask), as WanGP's _resize_video_mask makes it: the mask padded to whole clips of 17 frames by the last frame, the frames 0, 1, 5, 9 and 13 of each clip, the area nearest, and at least 0.5. Effects: fail. ### `reinjected` ```gizlang function reinjected(z: Vec, source: Vec, first: Vec, editable: Vec, sigma: f64, t: i64, frames: i64, area: i64): Vec ``` The video latents `z` ([24, t, lh, lw]) with the first `frames` latent frames of `source` put back in the noise `first` at `sigma` (WanGP's _reinject_video_source: torch.lerp in f32), but where `editable` (when given) is 1. Effects: fail. ## Methods ### `Arch.width` ```gizlang function Arch.width(self: Arch): i64 ``` Values of the attention's queries (and of its keys, and of its values) per row. Effects: fail. ### `Sequence.active` ```gizlang function Sequence.active(self: Sequence, queries: i64): Sequence ``` Sequence `s` where only its first `queries` rows are updated, the others (cached keys and values: RefSlot) being read only: the same positions and modulation rows, the runs cut to the first rows. Effects: fail. ### `Block.forward` ```gizlang function Block.forward(self: Block, a: Arch, x: Tensor, m: Modulation, s: Sequence): Tensor ``` The block over x ([rows, hidden], f32): x + gate (attention of the modulated norm), then the same with the MLP. On a GPU x is updated in place and returned; on the host a new tensor. Effects: fail, read, write, console, io. ### `Model.fill` ```gizlang function Model.fill(self: mut Model, sys: System, spare: i64): Result ``` Loads the blocks after those on the GPU while its free memory keeps `spare` bytes (as after the refiner leaves: `refiner = []` once the requests are made) and the two slots of the rest, which it stages (Staged). The blocks it holds on the GPU. Effects: fail, read, write, console, io. ### `Model.room` ```gizlang function Model.room(self: mut Model, need: i64): i64 ``` Stages blocks (the last on the GPU first) until the GPU has `need` bytes free, for a request whose rows take that much; `fill` brings them back. Returns how many it moved. Effects: fail, read, write, io. ### `Staged.on` ```gizlang function Staged.on(self: Staged, g: Gpu, slot: Buffer): Block ``` The staged block on the GPU memory `slot` (its bytes there once copied). Effects: fail. ### `Model.placed` ```gizlang function Model.placed(self: Model, t: Tensor): Tensor ``` `t` on the model's GPU (as it is when it has none). Effects: fail, read, write, console, io. ### `Model.tables` ```gizlang function Model.tables(self: Model, sys: System, times: Vec): Tables throws string ``` The modulation of the timesteps `times`: each AdaLN projection (Model.adaln) of their embeddings, once a schedule. A full model's AdaLN weights are about 40% of the checkpoint, so they load for this and leave. Effects: fail, read, write, console, io. ### `Model.prepare` ```gizlang function Model.prepare(self: mut Model, sys: System, steps: i64, shift: f64): Tables throws string ``` Prepares the schedule of `steps` at `shift` (scheduleTimes): its tables, made once and kept by the model, so the requests after (Model.schedule, Model.sample) do not make them. An option: kept tables hold memory where the model runs between requests (H3 at 6 steps: 339 MB on the GPU), so a long-running service prepares its schedules at start and a single request does not. Effects: fail, read, write, console, io. ### `Model.schedule` ```gizlang function Model.schedule(self: Model, sys: System, steps: i64, shift: f64): Tables throws string ``` The tables of the schedule of `steps` at `shift`: those Model.prepare keeps, else made for this request. Effects: fail, read, write, console, io. ### `Model.request` ```gizlang function Model.request(self: Model, l: Layout, text: Tensor, cond: Tensor, tags: Vec = [], condAudio: Vec = []): Request ``` The request of layout `l` with the text encoder's rows `text` ([l.text, textDim]) and their tags (TEXT, or VIDEO for the rows of a picture and its markers; none: all TEXT), the history's and keyframes' patch rows `cond` ([l.cond, latents x 4], noise-augmented as the caller wants) and the audio condition rows (`condAudio`: one [l.condAudio, 32], or none). Effects: fail, read, write, console, io. ### `Model.velocity` ```gizlang function Model.velocity(self: Model, sys: System, r: Request, tables: Tables, sigma: f64, audioSigma: f64, video: Tensor, audio: Tensor): Vec throws string ``` One pass with the video at `sigma` and the audio at `audioSigma`: the data-ward velocity (x0 = x + sigma v) of the video rows `video` ([l.video, latents x 4]) and audio rows `audio` ([l.audio, audio]), f32 where the model runs. Blocks not on the GPU copy from their staging to a slot while the block before runs. Effects: fail, read, write, console, io. ### `Model.sample` ```gizlang function Model.sample(self: Model, sys: System, r: Request, steps: i64, shift: f64, video: Tensor, audio: Tensor, cache: mut StepCache, from: i64 = 0, to: i64 = -1, solver: string = "euler", old: Vec = [], frozen: bool = false): Vec throws string ``` The clean rows of request `r` from the noise rows `video` and `audio`, in `steps` Euler passes over H3's two schedules (the video's shifted by `shift`, the audio's by AUDIO_SHIFT): [video, audio], f32 where the model runs. The schedule's tables are those Model.prepare keeps, else made for this request. `cache` is the caller's step cache (stepCache(0.0) runs every block, as production does: at 0.2, 2 of 6 passes skip and the video moves 15.7 dB from the full one). Passes `from` to `to` - 1 run (all by default), from rows at pass `from`, so a caller that runs them one at a time keeps one cache across them. `solver` is "euler" or "res_multistep" (WanGP's second order RES: the audio then moves on the video's schedule, which the last pass undoes; the result holds the denoised samples of the last pass after the two rows, and `old` is what a call before gave back of them). A `frozen` video stays as given, at the time of a clean condition (WanGP's target video condition). Effects: fail, read, write, console, io. ### `Decoder.decode` ```gizlang function Decoder.decode(self: Decoder, z0: Tensor): Tensor ``` The video of the normalized latents `z` ([latents, frames, h, w] on the host, as the transformer samples them): [3, pixel frames, 16 h, 16 w] RGB in [0, 1] on the host. Latent frames 5 n + 2 (n >= 1) give 17 n + 5 pixel frames. Effects: fail, read, write, console, io. ### `Encoder.fluxTokens` ```gizlang function Encoder.fluxTokens(self: Encoder, picture: Tensor): Tensor ``` FLUX's latent tokens ([h / 16 × w / 16, 128] f32 where the encoder runs, the grid's rows in order) of `picture` ([3, h, w] RGB in [-1, 1], h and w multiples of 16), as ComfyUI's VAE encode of BFL's VAE: the moments' means (the distribution's mode), patchified 2 × 2, batch-normed. The inverse of ImageDecoder.decode. Effects: fail, read, write, console, io. ### `Encoder.keyframe` ```gizlang function Encoder.keyframe(self: Encoder, picture: Tensor): Tensor ``` The normalized latents ([24, h / 16, w / 16], f32 on the host) of a keyframe `picture` ([3, h, w] RGB in [-1, 1], h and w multiples of 16), as WanGP's encode_condition makes them: the moments of tiles blended as the reference's _encode_clip, then the mean plus the deviation times torch's normal noise of seed 42, rounded to F16. Effects: fail, read, write, console, io. ### `Encoder.clip` ```gizlang function Encoder.clip(self: Encoder, video: Tensor, sample: bool = true, drawn: Vec = [], from: i64 = 0, total: i64 = 0): Tensor ``` The normalized latents ([24, latent frames, h / 16, w / 16], f32 on the host) of a clip `video` ([3, frames, h, w] RGB in [-1, 1], at most 17 frames: 1 + 4 k give 1 + k latent frames), as WanGP's encode_condition with keep_all_latents makes them for a continuation's history: the causal 3D CNN on every frame (each kernel's three temporal taps over the frame and the two before it, zeros before the first; levels 1 and 2 halve the frames too), tiles blended as keyframe's, then the mean plus the deviation times torch's normal noise of seed 42, rounded to F16 (the mean alone when not `sample`: the posterior's mode, as WanGP's encode makes a control video's latents; the normal noise `drawn` for all `total` latent frames of a longer video, from its frame `from`, when given). Effects: fail, read, write, console, io. ### `Encoder.video` ```gizlang function Encoder.video(self: Encoder, n: i64, frame: (i: i64) => Tensor): Tensor ``` The normalized latents ([24, 5 k - 3, h / 16, w / 16] for k clips, f32 on the host) of a control video of `n` frames, frame i being `frame(i)` ([3, h, w] RGB in [-1, 1]), as WanGP's vae.encode makes them: clips of 17 frames (the last frame repeated to fill the final clip), the posterior's mode of each, 5 latent frames a clip, the last 3 dropped. A clip's frames are made when the clip is encoded. Effects: fail, read, write, console, io. ### `Encoder.reference` ```gizlang function Encoder.reference(self: Encoder, n: i64, frame: (i: i64) => Tensor): Tensor ``` The latents of a reference video of `n` frames (17 k + 5; frame i is `frame(i)`, [3, h, w] RGB in [-1, 1]) as WanGP's encode_condition makes them without keep_all_latents: the moments of clips of 17 frames (the last frame repeated to fill the final clip) at 5 latent frames a clip, the last 3 dropped, the mean plus the deviation times one draw of torch's normal noise of seed 42 over all of them, rounded to F16. ([24, 5 k + 2, h / 16, w / 16]) Effects: fail, read, write, console, io. ### `Encoder.frozen` ```gizlang function Encoder.frozen(self: Encoder, n: i64, frame: (i: i64) => Tensor): Tensor ``` The latents of a video of `n` frames kept whole ([24, T, h / 16, w / 16], f32 on the host; clips of 17 frames and a last of the rest, which must be 1 + 4 k, give 1 + (frames - 1) / 4 latent frames each), as WanGP's encode_condition with keep_all_latents makes them for a frozen target video: the mean plus the deviation times one draw of torch's normal noise of seed 42 over all of them, rounded to F16. Effects: fail, read, write, console, io. ### `BigVgan.run` ```gizlang function BigVgan.run(self: BigVgan, x0: Tensor): Tensor ``` The generator's output for x ([t, c], time first, where it runs): [t times the rates' product, out], before any final clamp. Effects: fail, read, write, console, io. ### `AudioDecoder.decode` ```gizlang function AudioDecoder.decode(self: AudioDecoder, rows: Tensor): Tensor ``` The waveforms of the audio rows of a sample ([2 frames, 32]: the stereo channels one after the other, as H3 packs them): [2, frames x AUDIO_HOP] on the host, in [-1, 1]. Effects: fail, read, write, console, io. ### `AudioEncoder.encode` ```gizlang function AudioEncoder.encode(self: AudioEncoder, wave: Tensor): Tensor ``` The latent rows of `wave` ([2, samples] on the host: stereo, 32 kHz): [2 frames, 32] on the host, the channels one after the other as AudioDecoder.decode takes them; frames = samples / AUDIO_HOP, rounded up (zeros pad the end). Effects: fail, read, write, console, io. ### `Block.quantized` ```gizlang function Block.quantized(self: Block, bits: i64 = 8): Block ``` The block (BF16) with each linear quantized once as N.int8Rows quantizes x: its rows through ConvRot to INT8 over a scale a row (INT4 with 4 `bits`, where K.hasI4Mma: about 1.8 times the products' speed, at a loss of quality). Products then run as a W4 block's (linear). On the host, or on a GPU without INT8 matrix cores (K.hasI8Mma: Apple GPUs), the block stays BF16 and its products run on the F16 and BF16 matrix cores. Effects: fail, read, write, console, io. ### `Block.dequantized` ```gizlang function Block.dequantized(self: Block): Block ``` The block (W4 as stored, or INT8 as Block.quantized and ComfyUI's files hold it) as BF16: on a GPU without INT8 matrix cores, and before a requantization. Each linear is its INT8 weights times each output's scale, turned back from ConvRot's rotation: N.int8Rows takes each 256 inputs x to y = R x, so the weights of x are those of y times R. A LoRA stays as it is: its A must be BF16, not INT8. Effects: fail, read, write, console, io. ### `Block.parallel` ```gizlang function Block.parallel(self: Block, a: Arch, x: Tensor, m: Modulation, s: Sequence, slot: Option = undefined): Tensor ``` The parallel block over x ([rows, hidden], f32): x + gate (attention + SwiGLU MLP), both of one modulated LayerNorm. Its MLP's input linear has the gate and up rows interleaved (fluxBlock): row 2j is gate j, 2j + 1 up j. On a GPU x is updated in place and returned; on the host a new tensor. Effects: fail, read, write, console, io. ### `Flux.request` ```gizlang function Flux.request(self: Flux, text: Tensor, height: i64, width: i64, references: Vec = [], exact: bool = false): FluxRequest ``` The request of the text rows `text` (the text encoder's stacked hidden states, [text, textDim]) for an image of height by width tokens, edited from `references` (image to image; none: text to image). Reference i sits on RoPE layer -1 - i, the image on 0, the text's tokens on 0, 1, ... Effects: fail, read, write, console, io. ### `Flux.velocity` ```gizlang function Flux.velocity(self: Flux, r: FluxRequest, sigma: f64, next: f64, x: Tensor): Tensor ``` The velocity of the image tokens x ([rows, 128] f32) at sigma for the step to next. Effects: fail, read, write, console, io. ### `Flux.sample` ```gizlang function Flux.sample(self: Flux, r: FluxRequest, steps: i64, x0: Tensor, cache: f64 = 0.3, alpha: f64 = 16.0, refresh: i64 = 0): Tensor ``` Euler over FLUX.3's sigmas from noise x ([rows, 128] f32): the image tokens at sigma 0, with the step cache's `cache` limit (StepCache; 0 runs every block) and the timestep shift `alpha` (by default FLUX_SHIFT, 16). By default 0.3: FLUX.3 Klein at 1024² in 16 steps (4080 SUPER) samples in 2.86 s at 0, 2.01 s at 0.2 and 1.57 s at 0.3 with little change to the image; 0.1 skips none. INT4 blocks (loadFlux bits 4) move more from step to step and skip few. For an edit, `refresh` > 0 runs the reference images through the joint blocks only on steps 0, refresh, 2 refresh, ...: the other steps keep their keys and values (RefSlot) and update only the image and the text: faster, and an approximation, as the reference blocks take every step's target embedding. Effects: fail, read, write, console, io. ### `Flux.sampleWith` ```gizlang function Flux.sampleWith(self: Flux, r: FluxRequest, steps: i64, x0: Tensor, cache: f64, alpha: f64, refresh: i64, progress: (done: i64, total: i64) => bool): Tensor ``` Flux.sample that tells `progress` the steps done of `steps` after each (on a GPU, once it has run them), a service's progress and cancel: it returns false to stop, and the latents are then those so far. Effects: fail, read, write, console, io. ### `ImageDecoder.decode` ```gizlang function ImageDecoder.decode(self: ImageDecoder, tokens: Tensor, height: i64, width: i64): Tensor ``` Pixels ([3, 16 height, 16 width] in [-1, 1]) of FLUX latent tokens ([height × width, 4 channels] f32, the grid's rows in order): un-normed, unpatchified, then the decoder (in tiles above IMAGE_WHOLE tokens). Effects: fail, read, write, console, io. ### `TextEncoder.tokens` ```gizlang function TextEncoder.tokens(self: TextEncoder, prompt: string): Vec ``` The prompt's tokens in the chat template, before the padding. Effects: fail. ### `TextEncoder.encode` ```gizlang function TextEncoder.encode(self: mut TextEncoder, sys: System, prompt: string): Tensor throws string ``` The text rows of `prompt` for FLUX.3: [rows, 20480] f32 on the host. Effects: fail, read, write, console, io. ### `Upsampler.caption` ```gizlang function Upsampler.caption(self: mut Upsampler, sys: System, prompt: string, pictures: Vec = [], boxes: Vec = []): string throws string ``` The caption FLUX.3 Klein reads for `prompt`: text to image, or an edit of `pictures` (at most 10), laid out in `boxes` when there are any (their table follows the caption). Effects: fail, read, write, console, io. ### `Klein.condition` ```gizlang function Klein.condition(self: Klein, sys: System, prompt: string, pictures: Vec = [], size: Vec = [], megapixels: f64 = 1.0): KleinJob throws string ``` The job for `prompt`, edited from `pictures` (none: text to image). The image is `size` ([width, height] pixels, multiples of 16), else `megapixels` × 1024² at the first picture's aspect (a square without pictures), as the edit workflows' ImageScaleToTotalPixels sizes it: the first picture is scaled to it, and each picture is a reference (fluxReference). Fails where the prompt is too long for the text encoder. Effects: fail, read, write, console, io. ### `Klein.sample` ```gizlang function Klein.sample(self: Klein, job: KleinJob, seed: u64, steps: i64 = 16, cache: f64 = 0.3, refresh: i64 = 0): Tensor ``` The image's latent tokens ([height / 16 × width / 16, 128] f32) from seed `seed`'s noise (the same seed gives ComfyUI's image: fluxNoise), by Flux.sample with its `cache` and `refresh` at the timestep shift FLUX_SHIFT. Effects: fail, read, write, console, io. ### `Klein.sampleWith` ```gizlang function Klein.sampleWith(self: Klein, job: KleinJob, seed: u64, steps: i64, cache: f64, refresh: i64, progress: (done: i64, total: i64) => bool): Tensor ``` Klein.sample that tells `progress` the steps done (Flux.sampleWith). Effects: fail, read, write, console, io. ### `Klein.decode` ```gizlang function Klein.decode(self: Klein, job: KleinJob, z: Tensor): Img.Image ``` The image of latent tokens `z` (Klein.sample's). Effects: fail, read, write, console, io. ### `Klein.paint` ```gizlang function Klein.paint(self: Klein, sys: System, prompt: string, seed: u64, pictures: Vec = [], size: Vec = [], megapixels: f64 = 1.0, steps: i64 = 16, cache: f64 = 0.3, refresh: i64 = 0): Result ``` The image of `prompt` (Klein.condition's `pictures`, `size` and `megapixels`) from seed `seed`. Effects: fail, read, write, console, io. ### `H3Text.encode` ```gizlang function H3Text.encode(self: mut H3Text, sys: System, prompt: string, items: Vec = []): Conditioning throws string ``` The conditioning of `prompt` after `items` for H3, as giz's shared text encoder answers it (sharedText). Effects: fail, read, write, console, io. ### `ZImage.request` ```gizlang function ZImage.request(self: ZImage, text: Tensor, height: i64, width: i64): ZImageRequest ``` The request of `text` (ZImageText.encode's rows) for an image of `height` x `width` tokens (16 pixels each). Effects: fail, read, write, console, io. ### `ZImage.output` ```gizlang function ZImage.output(self: ZImage, r: ZImageRequest, sigma: f64, x: Tensor): Tensor ``` The minus velocity Z-Image gives for image tokens x ([height × width, 64] f32 where the model runs) at sigma. Effects: fail, read, write, console, io. ### `ZImage.sample` ```gizlang function ZImage.sample(self: ZImage, r: ZImageRequest, steps: i64, x0: Tensor, progress: (done: i64, total: i64) => bool): Tensor ``` The image tokens at sigma 0 from noise x ([height × width, 64] f32 on the host) over `steps` steps of ComfyUI's res_multistep (eta 0): Euler on the first and the last step, else the second-order multistep of the denoised now and before (arXiv 2308.02157). `progress` hears the steps done after each, and stops the sampling when it returns false. Effects: fail, read, write, console, io. ### `ZImageText.encode` ```gizlang function ZImageText.encode(self: mut ZImageText, sys: System, prompt: string): Tensor throws string ``` The text rows of `prompt` for Z-Image: [tokens, 2560] f32 on the host. Effects: fail, read, write, console, io. ### `ZImagePipeline.condition` ```gizlang function ZImagePipeline.condition(self: ZImagePipeline, sys: System, prompt: string, width: i64, height: i64): ZImageRequest throws string ``` The request of `prompt` for an image of `width` x `height` pixels (multiples of 16). Fails where the prompt is too long for the text encoder. Effects: fail, read, write, console, io. ### `ZImagePipeline.sampleWith` ```gizlang function ZImagePipeline.sampleWith(self: ZImagePipeline, r: ZImageRequest, seed: u64, steps: i64, progress: (done: i64, total: i64) => bool): Tensor ``` The image's tokens from seed `seed`'s noise (zImageNoise: ComfyUI's for the same seed) over `steps` steps; `progress` as ZImage.sample's. Effects: fail, read, write, console, io. ### `ZImagePipeline.decode` ```gizlang function ZImagePipeline.decode(self: ZImagePipeline, r: ZImageRequest, x: Tensor): Img.Image ``` The image of tokens `x` (ZImagePipeline.sampleWith's). Effects: fail, read, write, console, io. ### `H3.use` ```gizlang function H3.use(self: mut H3, sys: System, folder: string, steps: Vec): Result ``` Makes the transformer of model folder `folder` H3's from now on, as WanGP switches the DiT between H3's models (FL2VA, Ref2VA): the one before leaves the GPU and the page-locked memory first, so the two never take the card together; the VAEs and the text encoder stay. Nothing happens when `folder` is already H3's. Effects: fail, read, write, console, io. ### `H3.generate` ```gizlang function H3.generate(self: mut H3, sys: System, job: H3Job, progress: (stage: string, step: i64, steps: i64) => bool): Result ``` The video and audio of `job`, as WanGP's MiniMax H3 pipeline makes them for the same request and seed. `progress` hears each stage ("text", "keyframes", "sampling" with each pass, "decoding") with its step and steps, and answers whether to go on (false stops the job with "cancelled"). Effects: fail, read, write, console, io. ## Constants ### `HEAD` ```gizlang const HEAD = 128 ``` Values of an attention head of H3, and the rotary frequencies of each of its (t, h, w) axes. A head of any multiple of 32 values (`Arch.head`) rotates its first 3/4: the t, h and w angles of head / 8 frequencies each, twice (split halves). ### `AXIS_FREQUENCIES` ```gizlang const AXIS_FREQUENCIES = 16 ``` The rotary frequencies of each of a head's (t, h, w) axes: HEAD / 8. ### `QKV` ```gizlang const QKV = 0 ``` The index in Block.w of the fused query, key and value projection. ### `OUT` ```gizlang const OUT = 1 ``` The index in Block.w of the attention's output projection. ### `UP` ```gizlang const UP = 2 ``` The index in Block.w of the MLP's up projection (gate and up rows in pairs: gluRows). ### `DOWN` ```gizlang const DOWN = 3 ``` The index in Block.w of the MLP's down projection. ### `NORM1` ```gizlang const NORM1 = 4 ``` The index in Block.w of the norm weight before the attention. ### `NORM2` ```gizlang const NORM2 = 5 ``` The index in Block.w of the norm weight before the MLP. ### `Q_NORM` ```gizlang const Q_NORM = 6 ``` The index in Block.w of the queries' head norm weight. ### `K_NORM` ```gizlang const K_NORM = 7 ``` The index in Block.w of the keys' head norm weight. ### `POST_ATTN` ```gizlang const POST_ATTN = 8 ``` The indexes in Block.w of a sandwich block's norm weights after the attention and after the MLP (Z-Image): its outputs are normed before their gates. A block without them adds its gated outputs as they are. ### `POST_MLP` ```gizlang const POST_ATTN = 8; export const POST_MLP = 9 ``` ### `VIDEO` ```gizlang const VIDEO = 0 ``` The modalities of the rows of a sequence: each (timestep, modality) pair is a modulation row. ### `TEXT` ```gizlang const TEXT = 1 ``` The modality of text rows. ### `AUDIO` ```gizlang const AUDIO = 2 ``` The modality of audio rows. ### `KEYFRAME_TIME` ```gizlang const KEYFRAME_TIME = 0.999 ``` The timestep of keyframe rows at least (their noise augmentation). ### `AUDIO_CONDITION_TIME` ```gizlang const AUDIO_CONDITION_TIME = 1.0 ``` The timestep of audio condition rows and of a soundtrack's frames (clean audio; WanGP's AUDIO_COND_TIMESTEP). ### `AUDIO_SHIFT` ```gizlang const AUDIO_SHIFT = 3.0 ``` The shift of the audio's schedule. ### `AUDIO_HOP` ```gizlang const AUDIO_HOP = 800 ``` Samples of each audio latent frame (H3's audio is 40 latent frames a second). ### `AUDIO_SAMPLE_RATE` ```gizlang const AUDIO_SAMPLE_RATE = 32000 ``` Samples of a second of H3's audio. ### `FLUX_SHIFT` ```gizlang const FLUX_SHIFT = 16.0 ``` The timestep shift (alpha) of FLUX.3's sampling, as BFL's inference code (Pipeline.generate) and ComfyUI's klein workflows sample (its config.json says 6.93, which neither uses). ### `FLUX_IMAGE_ROWS` ```gizlang const FLUX_IMAGE_ROWS = 0 ``` The index in FluxRequest.sequences of the image's rows (the early blocks of the image stream). ### `FLUX_TEXT_ROWS` ```gizlang const FLUX_TEXT_ROWS = 1 ``` The index in FluxRequest.sequences of the text's rows (the early blocks of the text stream). ### `FLUX_JOINT` ```gizlang const FLUX_JOINT = 2 ``` The index in FluxRequest.sequences of the joint rows [image | text | references]. ### `FLUX_REFERENCE_ROWS` ```gizlang const FLUX_REFERENCE_ROWS = 3 ``` The index in FluxRequest.sequences of the references' rows alone. ### `FLUX_ACTIVE` ```gizlang const FLUX_ACTIVE = 4 ``` The index in FluxRequest.sequences of the joint rows that update the image and the text only. ### `IMAGE_WHOLE` ```gizlang const IMAGE_WHOLE = 6144 ``` The most latent tokens that ImageDecoder.decode decodes at once (1536 x 1024 pixels: its activations then fit a 16 GB card beside klein); a larger image is decoded in tiles of IMAGE_TILE tokens overlapping by IMAGE_OVERLAP, blended as H3's decoder's are (`fadedTiles`). ### `TEXT_LAYERS` ```gizlang const TEXT_LAYERS = 32 ``` The layers of Qwen3-VL-4B that FLUX.3 reads its text from (loadTextEncoder; convert them with gizml:llm). ### `H3_TEXT_PICTURE_ROOM` ```gizlang const H3_TEXT_PICTURE_ROOM = 805_306_368 ``` The device bytes the KV pool leaves free for the vision tower's pass over a picture (Vision.seen): 768 MiB, as a 1920 x 1088 picture's pass took 522 MiB on a 16 GB card. Without them the pool took the card and no picture fitted. ### `ZIMAGE_SHIFT` ```gizlang const ZIMAGE_SHIFT = 3.0 ``` The timestep shift of Z-Image's sampling (ComfyUI's ModelSamplingAuraFlow in its workflow). ### `ZIMAGE_TEXT_LAYERS` ```gizlang const ZIMAGE_TEXT_LAYERS = 35 ``` ### `ZIMAGE_STEPS` ```gizlang const ZIMAGE_STEPS = 8 ``` The steps Z-Image Turbo is made for (ComfyUI's workflow).