gizml:diffusion

video and image generation models (diffusion transformers), fast on any hardware.

A model family is a recipe: its config.json read as an Arch, and the names of its tensors. Every family runs on the same operations: the linears are gizml:tensor products on the matrix cores (BF16, or INT8 from a W4 checkpoint as production quantizes H3), and the row operations and attention are the kernels below. A GPU runs them; the host runs the same block in f32, which is the reference of the tests.

MiniMax H3 (the first family) is one single-stream transformer over a packed sequence of text, conditioning, audio and video rows. Each row has rotary (t, h, w) coordinates and a modulation row: the (timestep, modality) pair whose shift, scale and gate its block applies. Attention is full and unmasked over the whole sequence, so the order of the rows does not matter beyond the layout the caller keeps.

import * as Diffusion from "gizml:diffusion";

Types

Arch

type Arch = { family: string; hidden: i64; layers: i64; refiner: i64; heads: i64; head: i64; ffn: i64; latents: i64; audio: i64; patch: Vec<i64>; textDim: i64; freqDim: i64; timeHidden: i64; timeDim: i64; eps: f64; qkEps: f64; finalEps: f64; }

What a model computes, as its family's recipe reads config.json.

• family: string
• hidden: i64
• layers: i64
• refiner: i64
• heads: i64
• head: i64
• ffn: i64
• latents: i64
• audio: i64
• patch: Vec<i64>
• textDim: i64
• freqDim: i64
• timeHidden: i64
• timeDim: i64
• eps: f64
• qkEps: f64
• finalEps: f64

Block

type Block = { w: Vec<Tensor>; bias: Vec<Tensor>; codes: Vec<W4>; lora: Vec<Tensor>; }

One block's weights, by the indexes below: its 4 linears [out, in] as stored, then its 4 norms (f32). On a GPU a linear is BF16, or W4 when codes holds the decoding of each (w then holds its codes); on the host it is f32 values (W4: the code bytes as numbers). bias holds the linears' biases (f32, one per output) or none; lora the A ([rank, in]) and B ([out, rank]) of each linear's LoRA, applied beside it (BF16 on a GPU; a W4 block's A there is INT8 as N.int8Rows makes it, its row scales in B's columns), or none. UP's rows are the gate's and the up projection's in pairs (gluRows; SwiGLU: silu(gate) times up).

• w: Vec<M32.Tensor>
• bias: Vec<M32.Tensor>
• codes: Vec<M30.W4>
• lora: Vec<M32.Tensor>

Sequence

type Sequence = { rows: i64; map: Tensor; cos: Tensor; sin: Tensor; exact: bool; group: i64; runs: Vec<i64>; queries: i64; }

The rows of one packed sequence: per row its modulation row (map, as f32 values), and the cosines and sines of its rotary angles (per axis its frequencies: H3's t, h and w; FLUX.3's four). Tensors where the block runs. exact: its attention in BF16 where 8-bit matrix cores would take it (the reference of N.attentionRows quantized).

• rows: i64
• map: M32.Tensor
• cos: M32.Tensor
• sin: M32.Tensor
• exact: bool
• group: i64
• runs: Vec<i64>
• queries: i64

Modulation

type Modulation = { attn: Vec<Tensor>; mlp: Vec<Tensor>; }

A block's modulation: shift, scale and gate of the attention, then of the MLP, each [modulation rows, hidden] f32.

• attn: Vec<M32.Tensor>
• mlp: Vec<M32.Tensor>

Model

type Model = { arch: Arch; gpu: Option<Gpu>; grouped: bool; named: Map<string, Stored>; lora: Map<string, Stored>; curves: Tensor; map: Tensor; exact: bool; inverse: Vec<f32>; head: Vec<Tensor>; refiner: Vec<Block>; blocks: Vec<Block>; staged: Vec<Staged>; slots: Vec<Buffer>; kept: Vec<Tables>; }

A model ready to run: its Arch, where it runs, whether its blocks' linears are W4 (production's checkpoint) or BF16, the checkpoint's tensors and its LoRA's, the AdaLN curves of a pruned checkpoint (its table of time embeddings, [grid, rank] f32 on the host; none: the time embedder) and the LoRA's map to them (Model.adaln), the rotary inverse frequencies, the tensors outside the blocks, the token refiner, and the blocks that stay on the GPU (all of them when they fit; the others are staged in page-locked host memory and copy to the two slots ahead of their use). exact: attention in BF16 (Sequence). kept: the tables of the schedules it prepared (Model.prepare).

• arch: M20.Arch
• gpu: _gizlang_sdk_core.Option<M10.Gpu>
• grouped: bool
• named: _gizlang_sdk_collections.Map<string, gizlang_safetensors.Stored>
• lora: _gizlang_sdk_collections.Map<string, gizlang_safetensors.Stored>
• curves: M32.Tensor
• map: M32.Tensor
• exact: bool
• inverse: Vec<f32>
• head: Vec<M32.Tensor>
• refiner: Vec<M20.Block>
• blocks: Vec<M20.Block>
• staged: Vec<M20.Staged>
• slots: Vec<M10.Buffer>
• kept: Vec<M20.Tables>

Staged

type Staged = { host: Pinned; bytes: i64; parts: Vec<Part>; counts: Vec<i64>; }

A block in page-locked host memory: the bytes of its GPU tensors (blockParts) one after another, each from a multiple of 256, copied to a slot on the GPU ahead of each use (Gpu.copyAhead). counts: Block's lists' lengths.

• host: M10.Pinned
• bytes: i64
• parts: Vec<M20.Part>
• counts: Vec<i64>

Part

type Part = { shape: Vec<i64>; dtype: i64; at: i64; size: i64; host: Tensor; }

A tensor of a staged block: its shape, type, and bytes from at; a host tensor (at -1) stays as host.

• shape: Vec<i64>
• dtype: i64
• at: i64
• size: i64
• host: M32.Tensor

Tables

type Tables = { times: Vec<f64>; blocks: Vec<Modulation>; final: Vec<Tensor>; }

A schedule's modulation, where the model runs: for each block, the shift, scale and gate of its attention and MLP for each row 3 i + modality of the timesteps times; and the final layer's shift and scale for each timestep.

• times: Vec<f64>
• blocks: Vec<M20.Modulation>
• final: Vec<M32.Tensor>

Layout

type Layout = { text: i64; cond: i64; condAudio: i64; audio: i64; kept: i64; video: i64; positions: Vec<f64>; }

The rows of one request in packed order: text, keyframe video (conditioning), audio (the 2 channels of the audio frames, one channel after the other), then video (latent frames of a grid of patches); and the rotary (t, h, w) of each row.

• text: i64
• cond: i64
• condAudio: i64
• audio: i64
• kept: i64
• video: i64
• positions: Vec<f64>

Keyframe

type Keyframe = { at: f64; frames: i64; }

A keyframe: frames latent frames of the target's grid at pixel frame at of the target video.

• at: f64
• frames: i64

Conditions

type Conditions = { history: i64; audioHistory: i64; audioFirst: i64; kept: i64; }

What a request holds beside its keyframes (WanGP's continuation of a video and soundtrack): history latent frames of video before the target (the first condition rows), audioHistory latent audio frames before it and audioFirst at its start (the audio condition rows), and the target's first kept latent audio frames given (a soundtrack), which sampling keeps.

• history: i64
• audioHistory: i64
• audioFirst: i64
• kept: i64

RefRows

type RefRows = { kind: string; frames: i64; lh: i64; lw: i64; audio: i64; }

A reference of ref2va, in the order WanGP lists them (images, videos, audio): its kind ("image", "video" or "audio"), frames latent frames of lh x lw latents (patches of 2 x 2), and audio latent audio frames (a video's soundtrack or an audio reference). Its rows come after the keyframes' (video) and the audio conditions (audio).

• kind: string
• frames: i64
• lh: i64
• lw: i64
• audio: i64

Request

type Request = { layout: Layout; sequence: Sequence; text: Tensor; cond: Tensor; condAudio: Tensor; tags: Vec<i64>; }

A request ready to sample, where the model runs: its layout and sequence, its text rows after the token refiner, and its condition rows after the patch projections ([rows, hidden] f32): video, then audio.

• layout: M20.Layout
• sequence: M20.Sequence
• text: M32.Tensor
• cond: M32.Tensor
• condAudio: M32.Tensor
• tags: Vec<i64>

Conditioning

type Conditioning = { text: Tensor; tags: Vec<i64>; }

The text encoder's rows of a prompt: [rows, textDim] f32 on the host, and each row's tag (see Model.request).

• text: M32.Tensor
• tags: Vec<i64>

StepCache

type StepCache = { limit: f64; start: i64; first: Vec<Tensor>; rest: Vec<Tensor>; pending: Vec<Tensor>; step: i64; skipped: i64; }

A sampler's step cache (First Block Cache, as ParaAttention's), which Flux.sample and Model.sample share: where the first block's residual moved less than limit (its mean absolute change over its mean magnitude) since the last full step, a step adds that step's residual of the other blocks instead of running them. 0 runs every block; above 0 the output changes. A sampler calls cacheStart before its blocks, cacheSkip after the first and cacheEnd after the last. Steps before start run every block (WanGP's start step). first and rest hold the last full step's residuals, pending this step's first one until cacheEnd; step counts the steps, skipped those that reused them.

• limit: f64
• start: i64
• first: Vec<M32.Tensor>
• rest: Vec<M32.Tensor>
• pending: Vec<M32.Tensor>
• step: i64
• skipped: i64

Decoder

type Decoder = { arch: Arch; gpu: Option<Gpu>; blocks: Vec<Block>; gates: Vec<Modulation>; head: Vec<Tensor>; mean: Vec<f32>; std: Vec<f32>; inverse: Vec<f32>; }

The decoder of H3's video VAE (AutoencoderKLMiniMaxH3): each latent voxel of a clip is a token of a ViT (heads of 64, biases, a gate a channel for each branch, 4 register tokens and a zero token after the voxels, rotary angles of positions in [-1, 1) times 2 pi), and becomes 4 x 16 x 16 pixels. Videos decode in clips of 7 latent frames, 5 apart, and in tiles of 256 x 256 pixels that overlap by 64 at least, blended as the reference blends them.

• arch: M20.Arch
• gpu: _gizlang_sdk_core.Option<M10.Gpu>
• blocks: Vec<M20.Block>
• gates: Vec<M20.Modulation>
• head: Vec<M32.Tensor>
• mean: Vec<f32>
• std: Vec<f32>
• inverse: Vec<f32>

Encoder

type Encoder = { gpu: Option<Gpu>; named: Map<string, Tensor>; zeros: bool; mean: Vec<f32>; std: Vec<f32>; taps: Map<string, Tensor>; }

A VAE encoder (loadEncoder): its tensors by name (a 3 x 3 convolution as [out, in 9], BF16 on a GPU), where it runs, its padding (zeros: BFL's; else reflected: H3's), and the latents' mean and deviation by channel.

• gpu: _gizlang_sdk_core.Option<M10.Gpu>
• named: _gizlang_sdk_collections.Map<string, M32.Tensor>
• zeros: bool
• mean: Vec<f32>
• std: Vec<f32>
• taps: _gizlang_sdk_collections.Map<string, M32.Tensor>

AudioConv

type AudioConv = { w: Tensor; bias: Vec<Tensor>; }

A 1-D convolution: its weight ([out, in, k], or [in, out, k] transposed) and its bias, or none.

• w: M32.Tensor
• bias: Vec<M32.Tensor>

AudioSnake

type AudioSnake = { alpha: Tensor; inverse: Tensor; filters: Tensor; }

BigVGAN's alias-free SnakeBeta of channels channels: per channel e^alpha and 1 / (e^beta + 1e-9), and the 12 taps of the 2x upsampler then of the downsampler (filters [24] f32).

• alpha: M32.Tensor
• inverse: M32.Tensor
• filters: M32.Tensor

AudioBlock

type AudioBlock = { size: i64; first: Vec<AudioConv>; second: Vec<AudioConv>; snakes: Vec<AudioSnake>; }

One anti-aliased multi-periodicity block (AMPBlock1): for each dilation a dilated convolution and a plain one, each after its own Snake.

• size: i64
• first: Vec<M20.AudioConv>
• second: Vec<M20.AudioConv>
• snakes: Vec<M20.AudioSnake>

BigVgan

type BigVgan = { gpu: Option<Gpu>; pre: AudioConv; ups: Vec<AudioConv>; rates: Vec<i64>; blocks: Vec<AudioBlock>; dilations: Vec<i64>; post: AudioSnake; out: AudioConv; }

BigVGAN v2's generator (AMPBlock1, SnakeBeta): conv_pre (7 taps), then each stage a transposed convolution that upsamples by its rate and the mean of its residual blocks (kernels 3, 7, 11 by default, dilations 1, 3, 5), then the last Snake and conv_post (7 taps). H3's audio VAE decodes with one; LTX-2.5's vocoder and its bandwidth extension are two more.

• gpu: _gizlang_sdk_core.Option<M10.Gpu>
• pre: M20.AudioConv
• ups: Vec<M20.AudioConv>
• rates: Vec<i64>
• blocks: Vec<M20.AudioBlock>
• dilations: Vec<i64>
• post: M20.AudioSnake
• out: M20.AudioConv

AudioDecoder

type AudioDecoder = { stats: Vec<Tensor>; proj: AudioConv; vgan: BigVgan; }

H3's audio VAE decoder (AutoencoderKLMiniMaxH3Audio): latents of 32 channels to mono 32 kHz waveforms, one for each stereo channel. stats (the mean and deviation, [1, 32]) undo the latents' normalization (WanGP's MiniMaxH3AudioVAE), proj is the input projection, vgan the generator.

• stats: Vec<M32.Tensor>
• proj: M20.AudioConv
• vgan: M20.BigVgan

AudioEncoder

type AudioEncoder = { gpu: Option<Gpu>; convs: Map<string, AudioConv>; named: Map<string, Tensor>; }

H3's audio VAE encoder (AutoencoderKLMiniMaxH3Audio.encode, WanGP's MiniMaxH3AudioVAE): a mono 32 kHz waveform to latents of 32 channels, one frame a AUDIO_HOP samples. DAC's encoder: a convolution of 7 taps to 64 channels, then stages that double the channels (three residual units of Snake, a dilated convolution of 7 taps, Snake and a 1 x 1 convolution; Snake; a convolution of 2 stride taps at stride) to 2048, Snake and a convolution of 3 taps. Then the pre-block: a causal attention of 8 heads whose heads are averaged and pooled to 32 values, beside a projection of the layer-normed rows to 32; then a GeGLU MLP; then the mean projection, normalized as decode undoes it. The weights (weight norm applied) stay f32 on the host and go to gpu for each encode.

• gpu: _gizlang_sdk_core.Option<M10.Gpu>
• convs: _gizlang_sdk_collections.Map<string, M20.AudioConv>
• named: _gizlang_sdk_collections.Map<string, M32.Tensor>

Flux

type Flux = { arch: Arch; gpu: Option<Gpu>; time: Vec<Tensor>; target: Vec<Tensor>; early: Vec<Tensor>; single: Vec<Tensor>; imageIn: Tensor; referenceIn: Tensor; textIn: Tensor; finalMod: Tensor; finalOut: Tensor; textBlocks: Vec<Block>; imageBlocks: Vec<Block>; referenceBlocks: Vec<Block>; blocks: Vec<Block>; }

A FLUX.3 model where it runs: the time and target-time embedders (in, out), the modulation weights of the early and the joint streams (by mode), the input projections (image, reference images, text), the final layer's modulation and projection, and the blocks (text, image, reference, joint), their linears as Block.quantized makes them on a GPU.

• arch: M20.Arch
• gpu: _gizlang_sdk_core.Option<M10.Gpu>
• time: Vec<M32.Tensor>
• target: Vec<M32.Tensor>
• early: Vec<M32.Tensor>
• single: Vec<M32.Tensor>
• imageIn: M32.Tensor
• referenceIn: M32.Tensor
• textIn: M32.Tensor
• finalMod: M32.Tensor
• finalOut: M32.Tensor
• textBlocks: Vec<M20.Block>
• imageBlocks: Vec<M20.Block>
• referenceBlocks: Vec<M20.Block>
• blocks: Vec<M20.Block>

RefSlot

type RefSlot = { first: i64; count: i64; rows: Tensor; record: bool; }

A joint block's reference rows kept across steps (Flux.sample's refresh): their keys and values, [count, 2 width] BF16 in rows. A step that records copies them from rows first to first + count of the block's qkv; the others, which update only the rows before first, put them back there.

• first: i64
• count: i64
• rows: M32.Tensor
• record: bool

FluxReference

type FluxReference = { tokens: Tensor; height: i64; width: i64; }

A reference image of a FLUX.3 request (fluxReference): its latent tokens ([height × width, 128] f32, Encoder.fluxTokens) on a grid of height by width.

• tokens: M32.Tensor
• height: i64
• width: i64

FluxRequest

type FluxRequest = { text: Tensor; references: Vec<Tensor>; grid: Vec<i64>; sequences: Vec<Sequence>; }

A FLUX.3 request where the model runs: the text's rows ([text, hidden] f32, through txt_in), the reference images' rows (one [references, hidden] tensor through emb_in.image_cond, or none), the image's token grid (latent height / 2 by width / 2) and the sequences of its rows, by FLUX_IMAGE_ROWS and FLUX_TEXT_ROWS (the early blocks of those streams), FLUX_JOINT ([image | text | references]: the rows that change in a step come first) and, with references, FLUX_REFERENCE_ROWS (their rows alone) and FLUX_ACTIVE (the joint one updating the image and the text only, as the steps of Flux.sample's refresh run it). A record of the sequences themselves would exceed the compiler's limit.

• text: M32.Tensor
• references: Vec<M32.Tensor>
• grid: Vec<i64>
• sequences: Vec<M20.Sequence>

ImageDecoder

type ImageDecoder = { gpu: Option<Gpu>; named: Map<string, Tensor>; channels: i64; mean: Tensor; std: Tensor; }

BFL's image VAE decoder (loadImageDecoder): its tensors by name, where it runs, its latent channels, and the latent tokens' mean and std.

• gpu: _gizlang_sdk_core.Option<M10.Gpu>
• named: _gizlang_sdk_collections.Map<string, M32.Tensor>
• channels: i64
• mean: M32.Tensor
• std: M32.Tensor

TextEncoder

type TextEncoder = { engine: L.Engine; }

FLUX.3's text encoder (loadTextEncoder): Qwen3-VL-4B's first layers on a gizml:llm engine.

• engine: M23.Engine

Upsampler

type Upsampler = { engine: L.Engine; vision: L.Vision; prompts: JsonValue; }

FLUX.3's prompt upsampler (loadUpsampler): Qwen3.5's language model, its vision tower and the system prompts.

• engine: M23.Engine
• vision: M23.Vision
• prompts: gizlang_json.Json

FluxBox

type FluxBox = { bbox: Vec<i64>; desc: string; }

A box of a layout prompt: [y_min, x_min, y_max, x_max] on a 0-1000 canvas ((0, 0) at the top left) and what belongs in it.

• bbox: Vec<i64>
• desc: string

Klein

type Klein = { gpu: Gpu; flux: Ref<Flux>; text: Ref<TextEncoder>; encoder: Ref<Encoder>; decoder: Ref<ImageDecoder>; }

FLUX.3 klein ready to paint (loadKlein): its parts, each by reference (together they are more than a record holds) and already on gpu.

• gpu: M10.Gpu
• flux: Ref<M20.Flux>
• text: Ref<M20.TextEncoder>
• encoder: Ref<M20.Encoder>
• decoder: Ref<M20.ImageDecoder>

KleinJob

type KleinJob = { request: FluxRequest; width: i64; height: i64; }

What condition makes for sample and decode: the request and the image's size in pixels.

• request: M20.FluxRequest
• width: i64
• height: i64

H3Text

type H3Text = { engine: L.Engine; vision: L.Vision; }

H3's text encoder (loadH3Text): the language model and Qwen3-VL's vision tower.

• engine: M23.Engine
• vision: M23.Vision

Presented

type Presented = { kind: string; picture: Tensor; times: Vec<f64>; }

An item H3 shows before the prompt: "image" (picture [3, h, w] f32 in [0, 1] on the host, h and w multiples of 32), "video" (picture [n, 3, h, w], and times: each frame's second) or "audio" (its marker alone).

• kind: string
• picture: M32.Tensor
• times: Vec<f64>

ZImage

type ZImage = { arch: Arch; gpu: Option<Gpu>; time: Vec<Tensor>; capNorm: Tensor; capIn: Vec<Tensor>; imageIn: Vec<Tensor>; pads: Vec<Tensor>; mods: Vec<Vec<Tensor>>; finalMod: Vec<Tensor>; finalOut: Vec<Tensor>; context: Vec<Block>; noise: Vec<Block>; blocks: Vec<Block>; }

Z-Image where it runs: the time embedder's two linears and biases (on the host), the caption embedder's norm, linear and bias, the image embedder's linear and bias, the prompt's and the image's padding rows, each modulated block's modulation linear and bias (the noise refiners, then the blocks), the final layer's modulation and output (linear, bias each), and the blocks: context refiners, noise refiners and the 30, their linears as Block.quantized makes them on a GPU.

• arch: M20.Arch
• gpu: _gizlang_sdk_core.Option<M10.Gpu>
• time: Vec<M32.Tensor>
• capNorm: M32.Tensor
• capIn: Vec<M32.Tensor>
• imageIn: Vec<M32.Tensor>
• pads: Vec<M32.Tensor>
• mods: Vec<Vec<M32.Tensor>>
• finalMod: Vec<M32.Tensor>
• finalOut: Vec<M32.Tensor>
• context: Vec<M20.Block>
• noise: Vec<M20.Block>
• blocks: Vec<M20.Block>

ZImageRequest

type ZImageRequest = { prompt: Tensor; height: i64; width: i64; image: Sequence; joint: Sequence; }

What ZImage.request makes for the steps: the prompt's rows after the context refiners ([prompt rows padded, hidden] f32), the image's size in tokens, and the sequences of the image's rows (padded) and of [prompt; image].

• prompt: M32.Tensor
• height: i64
• width: i64
• image: M20.Sequence
• joint: M20.Sequence

ZImageText

type ZImageText = { engine: L.Engine; }

Z-Image's text encoder (loadZImageText): Qwen3-4B's first ZIMAGE_TEXT_LAYERS layers on a gizml:llm engine. The prompt in Qwen's chat template, with no padding, gives the residual stream after them (no final norm), as ComfyUI's Qwen3_4BModel (layer_idx -2) does.

• engine: M23.Engine

ZImagePipeline

type ZImagePipeline = { gpu: Gpu; model: Ref<ZImage>; text: Ref<ZImageText>; decoder: Ref<ImageDecoder>; }

Z-Image Turbo ready to paint (loadZImagePipeline): the model, its text encoder and FLUX.1's VAE decoder, each by reference and on gpu. As Klein, in stages a caller can time: condition, sample and decode.

• gpu: M10.Gpu
• model: Ref<M20.ZImage>
• text: Ref<M20.ZImageText>
• decoder: Ref<M20.ImageDecoder>

H3

type H3 = { model: Model; decoder: Decoder; encoder: Encoder; audio: AudioDecoder; listen: Vec<AudioEncoder>; text: Vec<H3Text>; textHost: string; textPort: i64; folder: string; spare: i64; }

H3 loaded for a service: the transformer and both VAEs on one GPU, and the text encoder: in this process (text, from the folder's text/ on the GPU) or giz's shared H3 text encoder service at textHost:textPort.

• model: M20.Model
• decoder: M20.Decoder
• encoder: M20.Encoder
• audio: M20.AudioDecoder
• listen: Vec<M20.AudioEncoder>
• text: Vec<M20.H3Text>
• textHost: string
• textPort: i64
• folder: string
• spare: i64

H3Job

type H3Job = { prompt: string; width: i64; height: i64; frames: i64; fps: i64; steps: i64; shift: f64; seed: i64; cache: f64; cacheStart: i64; start: Vec<Img.Image>; end: Vec<Img.Image>; injected: Vec<Img.Image>; positions: Vec<i64>; history: Vec<Tensor>; sound: Vec<Tensor>; soundtrack: bool; control: Vec<bytes>; mask: Vec<bytes>; denoising: f64; masking: f64; solver: string; frozen: bool; refs: Vec<H3Ref>; relative: f64; }

One request to H3, as giz's worker passes it on: the prompt; the canvas width x height (multiples of 32) and the frames (rounded up to 5 + 17 k) at fps; steps Euler passes at shift; the noise's seed; the step cache's limit and first step (stepCache; 0 runs every block); the first frame start and last frame end (none or one each) and the injected frames at positions (0-based frames; -1 the last). A first frame sets the video's size: the canvas's area at the picture's aspect, each side a multiple of 32 (WanGP's fit_canvas 0). history (none or one [3, n, height, width] in [-1, 1] on the host) continues a video: its frames come first, the last as the first frame, the others encoded (a sliding window's overlap, n = 1 + 17 k). sound (none or one [2, samples] at AUDIO_SAMPLE_RATE) is the audio from the history's start: the history's part continues the audio and, with soundtrack, the rest is kept as the target's (WanGP's audio prompt "A" or "K"). control (the frames of a control video from the window's first frame, each width * height * 3 RGB bytes) and mask (a byte of 0 or 1 a pixel of each frame of it, 1 for the area to make; none for no mask) edit that video (WanGP's video to video): it is encoded, and from the pass denoising (0 to 1) starts its latents are put back at each pass in the noise of that pass, but for the mask's area during the masking share of the passes.

• prompt: string
• width: i64
• height: i64
• frames: i64
• fps: i64
• steps: i64
• shift: f64
• seed: i64
• cache: f64
• cacheStart: i64
• start: Vec<gizlang_image.Image>
• end: Vec<gizlang_image.Image>
• injected: Vec<gizlang_image.Image>
• positions: Vec<i64>
• history: Vec<M32.Tensor>
• sound: Vec<M32.Tensor>
• soundtrack: bool
• control: Vec<bytes>
• mask: Vec<bytes>
• denoising: f64
• masking: f64
• solver: string
• frozen: bool
• refs: Vec<M20.H3Ref>
• relative: f64

H3Ref

type H3Ref = { kind: string; image: Vec<Img.Image>; frames: Vec<bytes>; width: i64; height: i64; sound: Vec<Tensor>; }

A reference of ref2va: an "image" (image, one; shown at relative per cent of the video's area), a "video" (its RGB frames, 17 k + 5 of them at width x height, multiples of 32; sound, its soundtrack when it has one) or an "audio" (sound, one wave [2, samples] at AUDIO_SAMPLE_RATE).

• kind: string
• image: Vec<gizlang_image.Image>
• frames: Vec<bytes>
• width: i64
• height: i64
• sound: Vec<M32.Tensor>

H3Output

type H3Output = { video: Tensor; audio: Tensor; }

A job's video ([3, frames, height, width] RGB in [0, 1] on the GPU) and its stereo audio ([2, samples] at AUDIO_SAMPLE_RATE on the host).

• video: M32.Tensor
• audio: M32.Tensor

Functions

archOf

function archOf(config: JsonValue): Arch throws string

The Arch of a model by its transformer's config.json; an error names what this module does not run yet.

Effects: fail.

loadBlock

function loadBlock(sys: System, named: Map<string, Stored>, lora: Map<string, Stored>, prefix: string, gpu: Option<Gpu>, interleaved: bool): Block throws string

Block prefix (as "blocks.3.") of the checkpoint's tensors on gpu when there is one, with the LoRA lora (an empty map: none): W4 or ComfyUI's INT8 ConvRot as stored when the checkpoint holds it so (w4Block, int8Block), else BF16 with the LoRA merged, its QKV rows a head at a time when interleaved (see weightAt; production's files hold them grouped). ComfyUI's scaled FP8 is quantized to INT8 (Block.quantized) after the merge: as BF16 it would take twice the GPU memory.

Effects: fail, read, write, console, io.

sequence

function sequence(positions: Vec<f64>, map: Vec<i64>, inverse: Vec<f32>, gpu: Option<Gpu>, exact: bool = false, group: i64 = 0): Sequence

The sequence of positions (rows of a position per axis: H3's t, h, w) whose rows use modulation rows map, with the rotary inverse frequencies inverse of each axis (H3's rope.inv_freq: head / 8 of them), on gpu when there is one. Rows attend within groups of group rows (0: all of them), as tiles of a VAE.

Effects: fail, read, write, console, io.

sequenceOf

function sequenceOf(positions: Vec<f64>, map: Vec<i64>, inverses: Vec<Vec<f32>>, gpu: Option<Gpu>, exact: bool = false, group: i64 = 0): Sequence

sequence with each axis its own inverse frequencies (Z-Image's 16, 24 and 24).

Effects: fail, read, write, console, io.

load

function load(sys: System, path: string, lora: string, gpu: Option<Gpu>, spare: i64, map: string = ""): Model throws string

The model at path: a transformer folder (config.json and its safetensors), or one .safetensors file of H3 (as production's W4 checkpoint), with the LoRA file lora unless it is "" (merged into BF16 weights, beside W4 ones), on gpu when there is one. Blocks stay on the GPU while its free memory keeps spare bytes for a pass (Model.fill stages the rest). A pruned checkpoint (AdaLN curves) takes a LoRA of the full AdaLN through map (WanGP's lora_affine_maps file of the model's table).

Effects: fail, read, write, console, io.

stagedBytes

function stagedBytes(b: Block): i64

The bytes of b's staged copy (Staged.bytes) without making it.

Effects: fail.

stage

function stage(g: Gpu, b: Block): Staged

Block b in page-locked host memory (Staged): its GPU tensors copied, its host ones kept.

Effects: fail, read, write, io.

sinusoids

function sinusoids(times: Vec<f64>, dim: i64): Tensor

Rows [cos | sin] of each time times the dim / 2 frequencies 10000^(-i / (dim / 2)), in f32 as diffusers' Timesteps (flip_sin_to_cos) computes them: the timestep embedding of H3 and FLUX.3.

Effects: fail.

noConditions

function noConditions(): Conditions

Conditions with no history and no kept audio.

layout

function layout(text: i64, frames: i64, height: i64, width: i64, audioFrames: i64, keyframes: Vec<Keyframe>, more: Conditions, refs: Vec<RefRows> = []): Layout

The rows of a request: text rows of text, the video's frames latent frames of height x width latents (patches of 2 x 2), audioFrames latent audio frames, and the keyframes and conditions. Time advances 1 a text row and a latent audio frame, and 5/3 a pixel frame; patches are on a grid of 32 around 16, scaled by the frame's aspect. The history comes first in time (from the text's end); the target, its keyframes and its audio after it.

Effects: fail.

sharedText

function sharedText(sys: System, host: string, port: i64, prompt: string, items: Vec<Presented> = []): Conditioning throws string

The conditioning of prompt after items (Presented) from a shared H3 text encoder at host:port: giz's wgp-api with H3_TEXT_ENCODER_SERVER (POST /api/h3/text-embeddings; it answers with the safetensors of "context" (1, rows, textDim) and "tags" (rows)). The request's bytes are those of its other clients, so its cache serves both.

Effects: fail, read, write, console, io.

scheduleTimes

function scheduleTimes(steps: i64, shift: f64): Vec<f64>

The timesteps of the passes of a schedule of steps at shift (Model.sample's): each pass's video, audio and keyframe timesteps, each once, then AUDIO_CONDITION_TIME.

Effects: fail.

sigmas

function sigmas(steps: i64, shift: f64): Vec<f64>

The sigmas of H3's flow schedule for steps passes: linspace(1, 0, steps + 1) shifted by shift (s σ / (1 + (s - 1) σ)), in f32 as the reference, without repeats.

Effects: fail.

resCoefficients

function resCoefficients(values: Vec<f64>): Vec<Vec<f64>>

WanGP's _res_multistep_coefficients: for each pass of the schedule values, the weights of the sample, of the denoised sample and of the pass before's (second order RES, arXiv:2308.02157); a first pass, and the last (to sigma 0), are Euler's.

Effects: fail.

stepCache

function stepCache(limit: f64, start: i64 = 0): StepCache

A step cache with LIMIT, which runs every block before step START.

patchify

function patchify(z: Tensor): Tensor

The video rows ([frames h w, latents x 4], f32 on the host) of latents [latents, frames, 2 h, 2 w] (unpatchify's inverse).

Effects: fail, read, write, console, io.

unpatchify

function unpatchify(rows: Tensor, latents: i64, frames: i64, h: i64, w: i64): Tensor

The video latents ([latents, frames, 2 h, 2 w], f32 on the host) of video rows [frames h w, latents x 4] (a patch of 2 x 2 latents a row, channel-major: column c 4 + 2 p + q holds channel c at (2 y + p, 2 x + q)).

Effects: fail, read, write, console, io.

loadDecoder

function loadDecoder(sys: System, folder: string, gpu: Option<Gpu>, quantized: bool = true): Decoder throws string

The decoder of a VAE folder (config.json and its safetensors, diffusers' names) or of WanGP's one .safetensors file (its names and H3's config; minimax_h3_video_vae_fp8mix holds E4M3 linears) on gpu when there is one, its linears BF16, or INT8 through ConvRot (Block.quantized) when quantized and the GPU has INT8 matrix cores and the widths allow it.

Effects: fail, read, write, console, io.

loadEncoder

function loadEncoder(sys: System, path: string, gpu: Option<Gpu>): Encoder throws string

The encoder of a VAE file on gpu when there is one: WanGP's H3 VAE (minimax_h3_video_vae_fp8mix: its encoder is BF16, its convolutions 3D) or BFL's FLUX.2 VAE (flux2-vae, BFL's names; its batch norm's statistics as the mean and deviation).

Effects: fail, read, write, console, io.

loadBigVgan

function loadBigVgan(sys: System, named: Map<string, Stored>, prefix: string, rates: Vec<i64>, kernels: Vec<i64>, dilations: Vec<i64>, numbered: bool, gpu: Option<Gpu>): BigVgan throws string

BigVGAN's generator under prefix of named (weight-normed or plain convolutions), its stages upsampling by rates, on gpu when there is one. H3's names (numbered) put each stage's transposed convolution under ups.i.0 and a block's Snakes under activations.0 to 5 in turn; LTX's under ups.i, and acts1.e and acts2.e.

Effects: fail, read, write, console, io.

loadAudioDecoder

function loadAudioDecoder(sys: System, path: string, gpu: Option<Gpu>): AudioDecoder throws string

The decoder of H3's audio VAE file (MiniMax-H3-audio_vae_fp32.safetensors), f32, on gpu when there is one.

Effects: fail, read, write, console, io.

loadAudioEncoder

function loadAudioEncoder(sys: System, path: string, gpu: Option<Gpu>): AudioEncoder throws string

The encoder of H3's audio VAE file (MiniMax-H3-audio_vae_fp32.safetensors), to run on gpu when there is one.

Effects: fail, io.

fluxArch

function fluxArch(): Arch

FLUX.3 Klein's sizes: BFL's single file holds no config, and these match its tensors (checked as it loads).

Effects: fail.

comfyInt8Rows

function comfyInt8Rows(sys: System, named: Map<string, Stored>, names: Vec<string>, gpu: Gpu): Vec<Tensor> throws string

Linears names of a ComfyUI INT8 ConvRot file (each name's .weight and .weight_scale: INT8 rows rotated by the regular Hadamard of 256, a scale a row) stacked by rows on gpu, as N.int8Rows' products read them (canonicalInto): [the INT8 weights [rows, in], their scales [1, rows] F32].

Effects: fail, read, write, console, io.

loadFlux

function loadFlux(sys: System, path: string, gpu: Option<Gpu>, bits: i64 = 8): Flux throws string

FLUX.3 Klein from its checkpoint (BFL's single .safetensors, BF16), on gpu when there is one: the blocks' linears INT8 (ConvRot), or INT4 with 4 bits where the GPU has INT4 matrix cores (K.hasI4Mma; else INT8): faster, at a loss of quality. A GPU without INT8 matrix cores keeps them BF16 (Block.quantized).

Effects: fail, read, write, console, io.

fluxSigmas

function fluxSigmas(steps: i64, shift: f64): Vec<f64>

FLUX.3's sigmas for steps steps: 1 to 1 / steps evenly, each shifted by shift (s σ / (1 + (s - 1) σ)), then 0.

Effects: fail.

fluxSize

function fluxSize(width: i64, height: i64, megapixels: f64, multiple: i64): Vec<i64>

The size (width, height) of megapixels × 1024² pixels at the aspect of width by height, each a multiple of multiple, as ComfyUI's ImageScaleToTotalPixels (an edit workflow's output: 4 megapixels, multiples of 16).

Effects: fail.

fluxReference

function fluxReference(e: Encoder, p: Img.Image, megapixels: f64 = 1.0): FluxReference

Picture p as a FLUX.3 reference image, as ComfyUI's ReferenceMultiLatent makes one: at megapixels × 1024² (its "resize", 1 by default; Pillow's LANCZOS), cut in the middle to multiples of 16 (VAE encode's crop) and encoded (Encoder.fluxTokens). Klein.condition first scales the first reference to the output's size, as the edit workflows do.

Effects: fail, read, write, console, io.

loadImageDecoder

function loadImageDecoder(sys: System, path: string, gpu: Option<Gpu>): ImageDecoder throws string

The decoder of BFL's VAE file (FLUX.2's or FLUX.1's ae, BFL's names), on gpu when there is one.

Effects: fail, read, write, console, io.

pixelsImage

function pixelsImage(x: Tensor): Img.Image

Pixels [3, h, w] in [-1, 1] (ImageDecoder.decode) as an RGB image: each (x + 1) × 127.5, clamped and rounded.

Effects: fail, read, write, console, io.

fluxNoise

function fluxNoise(seed: u64, height: i64, width: i64): Tensor

Seed seed's noise of FLUX latents of height × width tokens (an image of 16 × as many pixels) as ComfyUI draws it (torch.randn [1, 128, height, width] on the CPU), as tokens [height × width, 128]: a seed gives ComfyUI's image.

Effects: fail, read, write, console, io.

loadTextEncoder

function loadTextEncoder(sys: System, folder: string, gpu: Option<Gpu>): TextEncoder throws string

The text encoder of Qwen3-VL-4B converted by gizml:llm (L.convert with its first 32 layers), on gpu (none: the host).

Effects: fail, read, write, console, io.

parseFluxBoxes

function parseFluxBoxes(text: string): Vec<FluxBox> throws string

The boxes of a JSON list of {"bbox": [y0, x0, y1, x1], "desc": text} (the rows of ComfyUI's layout node).

Effects: fail.

convertUpsampler

function convertUpsampler(sys: System, from: string, to: string): i64 throws string

Converts BFL's flux-3-image-klein-pup folder to to: the language model by gizml:llm (4-bit layers, as ComfyUI's w4a8 file, and an 8-bit head) with its vision tower, and system_prompts.json. Returns the bytes written.

Effects: fail, read, write, console, io.

loadUpsampler

function loadUpsampler(sys: System, folder: string, gpu: Option<Gpu>): Upsampler throws string

The upsampler in folder (convertUpsampler's), on gpu (none: the host).

Effects: fail, read, write, console, io.

fluxBoxes

function fluxBoxes(boxes: Vec<FluxBox>): string

The boxes as Python's json.dumps writes them (ComfyUI's Flux3LayoutPrompt and BFL's bbox.py): rows of id ("obj_1", ...), bbox and desc.

Effects: fail.

fluxLayout

function fluxLayout(prompt: string, boxes: Vec<FluxBox>): string

The prompt of a layout without the upsampler: the boxes' table after the prompt (ComfyUI's Flux3LayoutPrompt).

Effects: fail.

textBits

function textBits(): L.Bits

The bits that FLUX.3's text encoder is converted with (convertTextEncoder; gizml:llm's L.converted takes them too).

convertTextEncoder

function convertTextEncoder(sys: System, from: string, to: string): i64 throws string

Converts Qwen3-VL-4B's Hugging Face folder from to the text encoder in folder to: its first TEXT_LAYERS layers by gizml:llm (textBits). Returns the bytes written.

Effects: fail, read, write, console, io.

loadKlein

function loadKlein(sys: System, model: string, vae: string, text: string, gpu: Gpu, bits: i64 = 8): Klein throws string

FLUX.3 klein on gpu: the model in BFL's single file model (loadFlux, with its bits), the VAE's in vae (BFL's ae, both the decoder and the encoder) and the text encoder in text (convertTextEncoder's folder). All parts take their buffers from this one Gpu, so its pool and its measures (Gpu.fits, Gpu.peak) hold all of them.

Effects: fail, read, write, console, io.

convertH3Text

function convertH3Text(sys: System, from: string, to: string): i64 throws string

Converts H3's text encoder (DeepBeepMeep's Qwen3-VL-32B-Instruct folder: config.json, the tokenizer and the layer-50 BF16 safetensors) to folder to by gizml:llm (4-bit layers, L.convert), with its vision tower. Returns the bytes written.

Effects: fail, read, write, console, io.

loadH3Text

function loadH3Text(sys: System, folder: string, gpu: Option<Gpu>, room: i64 = 805_306_368): H3Text throws string

H3's text encoder in folder (convertH3Text's), on gpu (none: the host), its KV pool leaving room bytes free for the vision tower's pass over a picture beside the card's reserve (by default H3_TEXT_PICTURE_ROOM).

Effects: fail, read, write, console, io.

serveH3Text

function serveH3Text(sys: System, listener: Listener, encoder: H3Text): i64 throws string

Serves encoder on listener as giz's shared H3 text encoder (wgp-api's; sharedText is a client): POST /api/h3/text-embeddings takes the form fields prompt, manifest (a JSON list of items: "image" or "video" with the name of its frames, [1, h, w, 3] ([n, h, w, 3] and "timestamps" for a video) in [0, 1], or "audio") and tensors (their safetensors), and answers the safetensors of "context" (1, rows, 5120) BF16 and "tags" (rows) I64; GET /v1/health answers whether it is ready. Requests take turns on the encoder.

Effects: fail, read, write, console, io.

zImageArch

function zImageArch(): Arch

Z-Image's sizes: ComfyUI's file holds no config, and these match its tensors (checked as it loads).

Effects: fail.

loadZImage

function loadZImage(sys: System, path: string, gpu: Option<Gpu>, bits: i64 = 8): ZImage throws string

Z-Image Turbo from ComfyUI's file (z_image_turbo_bf16.safetensors), on gpu when there is one: the blocks' linears INT8 (ConvRot), or INT4 with 4 bits where the GPU has INT4 matrix cores (K.hasI4Mma); BF16 on a GPU without INT8 matrix cores (Block.quantized).

Effects: fail, read, write, console, io.

zImageSigmas

function zImageSigmas(steps: i64, shift: f64): Vec<f64>

Z-Image's sigmas for steps steps, as ComfyUI's simple scheduler takes them from ModelSamplingAuraFlow's table of 1000 (multiplier 1): step k at t = (1000 - floor(k 1000 / steps)) / 1000 shifted by shift, then 0.

Effects: fail.

zImageNoise

function zImageNoise(seed: u64, height: i64, width: i64): Tensor

Seed seed's noise as Z-Image's tokens ([height × width, 64] f32 on the host): torch.randn of the latent [1, 16, 2 height, 2 width] on the CPU (ComfyUI's prepare_noise), patchified.

Effects: fail, read, write, console, io.

convertZImageText

function convertZImageText(sys: System, from: string, to: string): i64 throws string

Converts Qwen3-4B (a folder of its config.json, tokenizer files and BF16 safetensors, as ComfyUI's qwen_3_4b) to Z-Image's text encoder in folder to: its first ZIMAGE_TEXT_LAYERS layers by gizml:llm (textBits). Returns the bytes written.

Effects: fail, read, write, console, io.

loadZImageText

function loadZImageText(sys: System, folder: string, gpu: Option<Gpu>): ZImageText throws string

Z-Image's text encoder in folder (convertZImageText's), on gpu (none: the host).

Effects: fail, read, write, console, io.

zImageTokens

function zImageTokens(x: Tensor): Tensor

Z-Image's tokens ([height × width, 64], each (y, x, channel)) as ImageDecoder.decode reads them: (channel, y, x).

Effects: fail, read, write, console, io.

loadZImagePipeline

function loadZImagePipeline(sys: System, model: string, vae: string, text: string, gpu: Gpu, bits: i64 = 8): ZImagePipeline throws string

Z-Image Turbo on gpu from ComfyUI's model file (loadZImage, its linears of bits), the text encoder's folder (convertZImageText) and FLUX.1's ae.safetensors.

Effects: fail, read, write, console, io.

convertH3

function convertH3(sys: System, sources: Vec<string>, text: string, to: string): i64 throws string

Makes the model folder to from WanGP's files: sources are the transformer, its LoRA, the LoRA's AdaLN map, the video VAE and the audio VAE (h3Files' order), text the text encoder converted by convertH3Text ("" for none: the service then uses a shared one). Each file is linked where to is on the same file system, else copied.

Effects: fail, read, write, console, io.

loadH3

function loadH3(sys: System, folder: string, gpu: Gpu, text: string, spare: i64, steps: Vec<i64>): Result<H3, string>

H3 from model folder folder (convertH3) on gpu. text picks the text encoder: "local" (the folder's, in this process: about 13 GB more on the GPU) or "HOST:PORT" of a shared service (serveH3Text, or production's wgp-api). The tables of the schedules of steps (at H3's shift of 12; Model.prepare) are made first; then the transformer's blocks stay on the GPU while spare bytes stay free beside them, and the rest copy in at each pass.

Effects: fail, read, write, console, io.

h3Frames

function h3Frames(frames: i64): i64

The frames WanGP makes for a request of frames: at least 5, then up to 5 + 17 k.

Effects: fail.

latentMask

function latentMask(mask: Vec<bytes>, from: i64, n: i64, w: i64, h: i64, frames: i64, lh: i64, lw: i64): Vec<f32>

The mask of the source's frames latent frames ([frames, lh, lw] of 0 or 1) from mask (a byte of 0 or 1 a pixel of each frame, w x h; n frames from frame from on, and 1 past the mask), as WanGP's _resize_video_mask makes it: the mask padded to whole clips of 17 frames by the last frame, the frames 0, 1, 5, 9 and 13 of each clip, the area nearest, and at least 0.5.

Effects: fail.

reinjected

function reinjected(z: Vec<f32>, source: Vec<f32>, first: Vec<f32>, editable: Vec<f32>, sigma: f64, t: i64, frames: i64, area: i64): Vec<f32>

The video latents z ([24, t, lh, lw]) with the first frames latent frames of source put back in the noise first at sigma (WanGP's _reinject_video_source: torch.lerp in f32), but where editable (when given) is 1.

Effects: fail.

Methods

Arch.width

function Arch.width(self: Arch): i64

Values of the attention's queries (and of its keys, and of its values) per row.

Effects: fail.

Sequence.active

function Sequence.active(self: Sequence, queries: i64): Sequence

Sequence s where only its first queries rows are updated, the others (cached keys and values: RefSlot) being read only: the same positions and modulation rows, the runs cut to the first rows.

Effects: fail.

Block.forward

function Block.forward(self: Block, a: Arch, x: Tensor, m: Modulation, s: Sequence): Tensor

The block over x ([rows, hidden], f32): x + gate (attention of the modulated norm), then the same with the MLP. On a GPU x is updated in place and returned; on the host a new tensor.

Effects: fail, read, write, console, io.

Model.fill

function Model.fill(self: mut Model, sys: System, spare: i64): Result<i64, string>

Loads the blocks after those on the GPU while its free memory keeps spare bytes (as after the refiner leaves: refiner = [] once the requests are made) and the two slots of the rest, which it stages (Staged). The blocks it holds on the GPU.

Effects: fail, read, write, console, io.

Model.room

function Model.room(self: mut Model, need: i64): i64

Stages blocks (the last on the GPU first) until the GPU has need bytes free, for a request whose rows take that much; fill brings them back. Returns how many it moved.

Effects: fail, read, write, io.

Staged.on

function Staged.on(self: Staged, g: Gpu, slot: Buffer): Block

The staged block on the GPU memory slot (its bytes there once copied).

Effects: fail.

Model.placed

function Model.placed(self: Model, t: Tensor): Tensor

t on the model's GPU (as it is when it has none).

Effects: fail, read, write, console, io.

Model.tables

function Model.tables(self: Model, sys: System, times: Vec<f64>): Tables throws string

The modulation of the timesteps times: each AdaLN projection (Model.adaln) of their embeddings, once a schedule. A full model's AdaLN weights are about 40% of the checkpoint, so they load for this and leave.

Effects: fail, read, write, console, io.

Model.prepare

function Model.prepare(self: mut Model, sys: System, steps: i64, shift: f64): Tables throws string

Prepares the schedule of steps at shift (scheduleTimes): its tables, made once and kept by the model, so the requests after (Model.schedule, Model.sample) do not make them. An option: kept tables hold memory where the model runs between requests (H3 at 6 steps: 339 MB on the GPU), so a long-running service prepares its schedules at start and a single request does not.

Effects: fail, read, write, console, io.

Model.schedule

function Model.schedule(self: Model, sys: System, steps: i64, shift: f64): Tables throws string

The tables of the schedule of steps at shift: those Model.prepare keeps, else made for this request.

Effects: fail, read, write, console, io.

Model.request

function Model.request(self: Model, l: Layout, text: Tensor, cond: Tensor, tags: Vec<i64> = [], condAudio: Vec<Tensor> = []): Request

The request of layout l with the text encoder's rows text ([l.text, textDim]) and their tags (TEXT, or VIDEO for the rows of a picture and its markers; none: all TEXT), the history's and keyframes' patch rows cond ([l.cond, latents x 4], noise-augmented as the caller wants) and the audio condition rows (condAudio: one [l.condAudio, 32], or none).

Effects: fail, read, write, console, io.

Model.velocity

function Model.velocity(self: Model, sys: System, r: Request, tables: Tables, sigma: f64, audioSigma: f64, video: Tensor, audio: Tensor): Vec<Tensor> throws string

One pass with the video at sigma and the audio at audioSigma: the data-ward velocity (x0 = x + sigma v) of the video rows video ([l.video, latents x 4]) and audio rows audio ([l.audio, audio]), f32 where the model runs. Blocks not on the GPU copy from their staging to a slot while the block before runs.

Effects: fail, read, write, console, io.

Model.sample

function Model.sample(self: Model, sys: System, r: Request, steps: i64, shift: f64, video: Tensor, audio: Tensor, cache: mut StepCache, from: i64 = 0, to: i64 = -1, solver: string = "euler", old: Vec<Tensor> = [], frozen: bool = false): Vec<Tensor> throws string

The clean rows of request r from the noise rows video and audio, in steps Euler passes over H3's two schedules (the video's shifted by shift, the audio's by AUDIO_SHIFT): [video, audio], f32 where the model runs. The schedule's tables are those Model.prepare keeps, else made for this request. cache is the caller's step cache (stepCache(0.0) runs every block, as production does: at 0.2, 2 of 6 passes skip and the video moves 15.7 dB from the full one). Passes from to to - 1 run (all by default), from rows at pass from, so a caller that runs them one at a time keeps one cache across them. solver is "euler" or "res_multistep" (WanGP's second order RES: the audio then moves on the video's schedule, which the last pass undoes; the result holds the denoised samples of the last pass after the two rows, and old is what a call before gave back of them). A frozen video stays as given, at the time of a clean condition (WanGP's target video condition).

Effects: fail, read, write, console, io.

Decoder.decode

function Decoder.decode(self: Decoder, z0: Tensor): Tensor

The video of the normalized latents z ([latents, frames, h, w] on the host, as the transformer samples them): [3, pixel frames, 16 h, 16 w] RGB in [0, 1] on the host. Latent frames 5 n + 2 (n >= 1) give 17 n + 5 pixel frames.

Effects: fail, read, write, console, io.

Encoder.fluxTokens

function Encoder.fluxTokens(self: Encoder, picture: Tensor): Tensor

FLUX's latent tokens ([h / 16 × w / 16, 128] f32 where the encoder runs, the grid's rows in order) of picture ([3, h, w] RGB in [-1, 1], h and w multiples of 16), as ComfyUI's VAE encode of BFL's VAE: the moments' means (the distribution's mode), patchified 2 × 2, batch-normed. The inverse of ImageDecoder.decode.

Effects: fail, read, write, console, io.

Encoder.keyframe

function Encoder.keyframe(self: Encoder, picture: Tensor): Tensor

The normalized latents ([24, h / 16, w / 16], f32 on the host) of a keyframe picture ([3, h, w] RGB in [-1, 1], h and w multiples of 16), as WanGP's encode_condition makes them: the moments of tiles blended as the reference's _encode_clip, then the mean plus the deviation times torch's normal noise of seed 42, rounded to F16.

Effects: fail, read, write, console, io.

Encoder.clip

function Encoder.clip(self: Encoder, video: Tensor, sample: bool = true, drawn: Vec<f32> = [], from: i64 = 0, total: i64 = 0): Tensor

The normalized latents ([24, latent frames, h / 16, w / 16], f32 on the host) of a clip video ([3, frames, h, w] RGB in [-1, 1], at most 17 frames: 1 + 4 k give 1 + k latent frames), as WanGP's encode_condition with keep_all_latents makes them for a continuation's history: the causal 3D CNN on every frame (each kernel's three temporal taps over the frame and the two before it, zeros before the first; levels 1 and 2 halve the frames too), tiles blended as keyframe's, then the mean plus the deviation times torch's normal noise of seed 42, rounded to F16 (the mean alone when not sample: the posterior's mode, as WanGP's encode makes a control video's latents; the normal noise drawn for all total latent frames of a longer video, from its frame from, when given).

Effects: fail, read, write, console, io.

Encoder.video

function Encoder.video(self: Encoder, n: i64, frame: (i: i64) => Tensor): Tensor

The normalized latents ([24, 5 k - 3, h / 16, w / 16] for k clips, f32 on the host) of a control video of n frames, frame i being frame(i) ([3, h, w] RGB in [-1, 1]), as WanGP's vae.encode makes them: clips of 17 frames (the last frame repeated to fill the final clip), the posterior's mode of each, 5 latent frames a clip, the last 3 dropped. A clip's frames are made when the clip is encoded.

Effects: fail, read, write, console, io.

Encoder.reference

function Encoder.reference(self: Encoder, n: i64, frame: (i: i64) => Tensor): Tensor

The latents of a reference video of n frames (17 k + 5; frame i is frame(i), [3, h, w] RGB in [-1, 1]) as WanGP's encode_condition makes them without keep_all_latents: the moments of clips of 17 frames (the last frame repeated to fill the final clip) at 5 latent frames a clip, the last 3 dropped, the mean plus the deviation times one draw of torch's normal noise of seed 42 over all of them, rounded to F16. ([24, 5 k + 2, h / 16, w / 16])

Effects: fail, read, write, console, io.

Encoder.frozen

function Encoder.frozen(self: Encoder, n: i64, frame: (i: i64) => Tensor): Tensor

The latents of a video of n frames kept whole ([24, T, h / 16, w / 16], f32 on the host; clips of 17 frames and a last of the rest, which must be 1 + 4 k, give 1 + (frames - 1) / 4 latent frames each), as WanGP's encode_condition with keep_all_latents makes them for a frozen target video: the mean plus the deviation times one draw of torch's normal noise of seed 42 over all of them, rounded to F16.

Effects: fail, read, write, console, io.

BigVgan.run

function BigVgan.run(self: BigVgan, x0: Tensor): Tensor

The generator's output for x ([t, c], time first, where it runs): [t times the rates' product, out], before any final clamp.

Effects: fail, read, write, console, io.

AudioDecoder.decode

function AudioDecoder.decode(self: AudioDecoder, rows: Tensor): Tensor

The waveforms of the audio rows of a sample ([2 frames, 32]: the stereo channels one after the other, as H3 packs them): [2, frames x AUDIO_HOP] on the host, in [-1, 1].

Effects: fail, read, write, console, io.

AudioEncoder.encode

function AudioEncoder.encode(self: AudioEncoder, wave: Tensor): Tensor

The latent rows of wave ([2, samples] on the host: stereo, 32 kHz): [2 frames, 32] on the host, the channels one after the other as AudioDecoder.decode takes them; frames = samples / AUDIO_HOP, rounded up (zeros pad the end).

Effects: fail, read, write, console, io.

Block.quantized

function Block.quantized(self: Block, bits: i64 = 8): Block

The block (BF16) with each linear quantized once as N.int8Rows quantizes x: its rows through ConvRot to INT8 over a scale a row (INT4 with 4 bits, where K.hasI4Mma: about 1.8 times the products' speed, at a loss of quality). Products then run as a W4 block's (linear). On the host, or on a GPU without INT8 matrix cores (K.hasI8Mma: Apple GPUs), the block stays BF16 and its products run on the F16 and BF16 matrix cores.

Effects: fail, read, write, console, io.

Block.dequantized

function Block.dequantized(self: Block): Block

The block (W4 as stored, or INT8 as Block.quantized and ComfyUI's files hold it) as BF16: on a GPU without INT8 matrix cores, and before a requantization. Each linear is its INT8 weights times each output's scale, turned back from ConvRot's rotation: N.int8Rows takes each 256 inputs x to y = R x, so the weights of x are those of y times R. A LoRA stays as it is: its A must be BF16, not INT8.

Effects: fail, read, write, console, io.

Block.parallel

function Block.parallel(self: Block, a: Arch, x: Tensor, m: Modulation, s: Sequence, slot: Option<RefSlot> = undefined): Tensor

The parallel block over x ([rows, hidden], f32): x + gate (attention + SwiGLU MLP), both of one modulated LayerNorm. Its MLP's input linear has the gate and up rows interleaved (fluxBlock): row 2j is gate j, 2j + 1 up j. On a GPU x is updated in place and returned; on the host a new tensor.

Effects: fail, read, write, console, io.

Flux.request

function Flux.request(self: Flux, text: Tensor, height: i64, width: i64, references: Vec<FluxReference> = [], exact: bool = false): FluxRequest

The request of the text rows text (the text encoder's stacked hidden states, [text, textDim]) for an image of height by width tokens, edited from references (image to image; none: text to image). Reference i sits on RoPE layer -1 - i, the image on 0, the text's tokens on 0, 1, ...

Effects: fail, read, write, console, io.

Flux.velocity

function Flux.velocity(self: Flux, r: FluxRequest, sigma: f64, next: f64, x: Tensor): Tensor

The velocity of the image tokens x ([rows, 128] f32) at sigma for the step to next.

Effects: fail, read, write, console, io.

Flux.sample

function Flux.sample(self: Flux, r: FluxRequest, steps: i64, x0: Tensor, cache: f64 = 0.3, alpha: f64 = 16.0, refresh: i64 = 0): Tensor

Euler over FLUX.3's sigmas from noise x ([rows, 128] f32): the image tokens at sigma 0, with the step cache's cache limit (StepCache; 0 runs every block) and the timestep shift alpha (by default FLUX_SHIFT, 16). By default 0.3: FLUX.3 Klein at 1024² in 16 steps (4080 SUPER) samples in 2.86 s at 0, 2.01 s at 0.2 and 1.57 s at 0.3 with little change to the image; 0.1 skips none. INT4 blocks (loadFlux bits 4) move more from step to step and skip few. For an edit, refresh > 0 runs the reference images through the joint blocks only on steps 0, refresh, 2 refresh, ...: the other steps keep their keys and values (RefSlot) and update only the image and the text: faster, and an approximation, as the reference blocks take every step's target embedding.

Effects: fail, read, write, console, io.

Flux.sampleWith

function Flux.sampleWith(self: Flux, r: FluxRequest, steps: i64, x0: Tensor, cache: f64, alpha: f64, refresh: i64, progress: (done: i64, total: i64) => bool): Tensor

Flux.sample that tells progress the steps done of steps after each (on a GPU, once it has run them), a service's progress and cancel: it returns false to stop, and the latents are then those so far.

Effects: fail, read, write, console, io.

ImageDecoder.decode

function ImageDecoder.decode(self: ImageDecoder, tokens: Tensor, height: i64, width: i64): Tensor

Pixels ([3, 16 height, 16 width] in [-1, 1]) of FLUX latent tokens ([height × width, 4 channels] f32, the grid's rows in order): un-normed, unpatchified, then the decoder (in tiles above IMAGE_WHOLE tokens).

Effects: fail, read, write, console, io.

TextEncoder.tokens

function TextEncoder.tokens(self: TextEncoder, prompt: string): Vec<i64>

The prompt's tokens in the chat template, before the padding.

Effects: fail.

TextEncoder.encode

function TextEncoder.encode(self: mut TextEncoder, sys: System, prompt: string): Tensor throws string

The text rows of prompt for FLUX.3: [rows, 20480] f32 on the host.

Effects: fail, read, write, console, io.

Upsampler.caption

function Upsampler.caption(self: mut Upsampler, sys: System, prompt: string, pictures: Vec<Img.Image> = [], boxes: Vec<FluxBox> = []): string throws string

The caption FLUX.3 Klein reads for prompt: text to image, or an edit of pictures (at most 10), laid out in boxes when there are any (their table follows the caption).

Effects: fail, read, write, console, io.

Klein.condition

function Klein.condition(self: Klein, sys: System, prompt: string, pictures: Vec<Img.Image> = [], size: Vec<i64> = [], megapixels: f64 = 1.0): KleinJob throws string

The job for prompt, edited from pictures (none: text to image). The image is size ([width, height] pixels, multiples of 16), else megapixels × 1024² at the first picture's aspect (a square without pictures), as the edit workflows' ImageScaleToTotalPixels sizes it: the first picture is scaled to it, and each picture is a reference (fluxReference). Fails where the prompt is too long for the text encoder.

Effects: fail, read, write, console, io.

Klein.sample

function Klein.sample(self: Klein, job: KleinJob, seed: u64, steps: i64 = 16, cache: f64 = 0.3, refresh: i64 = 0): Tensor

The image's latent tokens ([height / 16 × width / 16, 128] f32) from seed seed's noise (the same seed gives ComfyUI's image: fluxNoise), by Flux.sample with its cache and refresh at the timestep shift FLUX_SHIFT.

Effects: fail, read, write, console, io.

Klein.sampleWith

function Klein.sampleWith(self: Klein, job: KleinJob, seed: u64, steps: i64, cache: f64, refresh: i64, progress: (done: i64, total: i64) => bool): Tensor

Klein.sample that tells progress the steps done (Flux.sampleWith).

Effects: fail, read, write, console, io.

Klein.decode

function Klein.decode(self: Klein, job: KleinJob, z: Tensor): Img.Image

The image of latent tokens z (Klein.sample's).

Effects: fail, read, write, console, io.

Klein.paint

function Klein.paint(self: Klein, sys: System, prompt: string, seed: u64, pictures: Vec<Img.Image> = [], size: Vec<i64> = [], megapixels: f64 = 1.0, steps: i64 = 16, cache: f64 = 0.3, refresh: i64 = 0): Result<Img.Image, string>

The image of prompt (Klein.condition's pictures, size and megapixels) from seed seed.

Effects: fail, read, write, console, io.

H3Text.encode

function H3Text.encode(self: mut H3Text, sys: System, prompt: string, items: Vec<Presented> = []): Conditioning throws string

The conditioning of prompt after items for H3, as giz's shared text encoder answers it (sharedText).

Effects: fail, read, write, console, io.

ZImage.request

function ZImage.request(self: ZImage, text: Tensor, height: i64, width: i64): ZImageRequest

The request of text (ZImageText.encode's rows) for an image of height x width tokens (16 pixels each).

Effects: fail, read, write, console, io.

ZImage.output

function ZImage.output(self: ZImage, r: ZImageRequest, sigma: f64, x: Tensor): Tensor

The minus velocity Z-Image gives for image tokens x ([height × width, 64] f32 where the model runs) at sigma.

Effects: fail, read, write, console, io.

ZImage.sample

function ZImage.sample(self: ZImage, r: ZImageRequest, steps: i64, x0: Tensor, progress: (done: i64, total: i64) => bool): Tensor

The image tokens at sigma 0 from noise x ([height × width, 64] f32 on the host) over steps steps of ComfyUI's res_multistep (eta 0): Euler on the first and the last step, else the second-order multistep of the denoised now and before (arXiv 2308.02157). progress hears the steps done after each, and stops the sampling when it returns false.

Effects: fail, read, write, console, io.

ZImageText.encode

function ZImageText.encode(self: mut ZImageText, sys: System, prompt: string): Tensor throws string

The text rows of prompt for Z-Image: [tokens, 2560] f32 on the host.

Effects: fail, read, write, console, io.

ZImagePipeline.condition

function ZImagePipeline.condition(self: ZImagePipeline, sys: System, prompt: string, width: i64, height: i64): ZImageRequest throws string

The request of prompt for an image of width x height pixels (multiples of 16). Fails where the prompt is too long for the text encoder.

Effects: fail, read, write, console, io.

ZImagePipeline.sampleWith

function ZImagePipeline.sampleWith(self: ZImagePipeline, r: ZImageRequest, seed: u64, steps: i64, progress: (done: i64, total: i64) => bool): Tensor

The image's tokens from seed seed's noise (zImageNoise: ComfyUI's for the same seed) over steps steps; progress as ZImage.sample's.

Effects: fail, read, write, console, io.

ZImagePipeline.decode

function ZImagePipeline.decode(self: ZImagePipeline, r: ZImageRequest, x: Tensor): Img.Image

The image of tokens x (ZImagePipeline.sampleWith's).

Effects: fail, read, write, console, io.

H3.use

function H3.use(self: mut H3, sys: System, folder: string, steps: Vec<i64>): Result<bool, string>

Makes the transformer of model folder folder H3's from now on, as WanGP switches the DiT between H3's models (FL2VA, Ref2VA): the one before leaves the GPU and the page-locked memory first, so the two never take the card together; the VAEs and the text encoder stay. Nothing happens when folder is already H3's.

Effects: fail, read, write, console, io.

H3.generate

function H3.generate(self: mut H3, sys: System, job: H3Job, progress: (stage: string, step: i64, steps: i64) => bool): Result<H3Output, string>

The video and audio of job, as WanGP's MiniMax H3 pipeline makes them for the same request and seed. progress hears each stage ("text", "keyframes", "sampling" with each pass, "decoding") with its step and steps, and answers whether to go on (false stops the job with "cancelled").

Effects: fail, read, write, console, io.

Constants

HEAD

const HEAD = 128

Values of an attention head of H3, and the rotary frequencies of each of its (t, h, w) axes. A head of any multiple of 32 values (Arch.head) rotates its first 3/4: the t, h and w angles of head / 8 frequencies each, twice (split halves).

AXIS_FREQUENCIES

const AXIS_FREQUENCIES = 16

The rotary frequencies of each of a head's (t, h, w) axes: HEAD / 8.

QKV

const QKV = 0

The index in Block.w of the fused query, key and value projection.

OUT

const OUT = 1

The index in Block.w of the attention's output projection.

UP

const UP = 2

The index in Block.w of the MLP's up projection (gate and up rows in pairs: gluRows).

DOWN

const DOWN = 3

The index in Block.w of the MLP's down projection.

NORM1

const NORM1 = 4

The index in Block.w of the norm weight before the attention.

NORM2

const NORM2 = 5

The index in Block.w of the norm weight before the MLP.

Q_NORM

const Q_NORM = 6

The index in Block.w of the queries' head norm weight.

K_NORM

const K_NORM = 7

The index in Block.w of the keys' head norm weight.

POST_ATTN

const POST_ATTN = 8

The indexes in Block.w of a sandwich block's norm weights after the attention and after the MLP (Z-Image): its outputs are normed before their gates. A block without them adds its gated outputs as they are.

POST_MLP

const POST_ATTN = 8; export const POST_MLP = 9

VIDEO

const VIDEO = 0

The modalities of the rows of a sequence: each (timestep, modality) pair is a modulation row.

TEXT

const TEXT = 1

The modality of text rows.

AUDIO

const AUDIO = 2

The modality of audio rows.

KEYFRAME_TIME

const KEYFRAME_TIME = 0.999

The timestep of keyframe rows at least (their noise augmentation).

AUDIO_CONDITION_TIME

const AUDIO_CONDITION_TIME = 1.0

The timestep of audio condition rows and of a soundtrack's frames (clean audio; WanGP's AUDIO_COND_TIMESTEP).

AUDIO_SHIFT

const AUDIO_SHIFT = 3.0

The shift of the audio's schedule.

AUDIO_HOP

const AUDIO_HOP = 800

Samples of each audio latent frame (H3's audio is 40 latent frames a second).

AUDIO_SAMPLE_RATE

const AUDIO_SAMPLE_RATE = 32000

Samples of a second of H3's audio.

FLUX_SHIFT

const FLUX_SHIFT = 16.0

The timestep shift (alpha) of FLUX.3's sampling, as BFL's inference code (Pipeline.generate) and ComfyUI's klein workflows sample (its config.json says 6.93, which neither uses).

FLUX_IMAGE_ROWS

const FLUX_IMAGE_ROWS = 0

The index in FluxRequest.sequences of the image's rows (the early blocks of the image stream).

FLUX_TEXT_ROWS

const FLUX_TEXT_ROWS = 1

The index in FluxRequest.sequences of the text's rows (the early blocks of the text stream).

FLUX_JOINT

const FLUX_JOINT = 2

The index in FluxRequest.sequences of the joint rows [image | text | references].

FLUX_REFERENCE_ROWS

const FLUX_REFERENCE_ROWS = 3

The index in FluxRequest.sequences of the references' rows alone.

FLUX_ACTIVE

const FLUX_ACTIVE = 4

The index in FluxRequest.sequences of the joint rows that update the image and the text only.

IMAGE_WHOLE

const IMAGE_WHOLE = 6144

The most latent tokens that ImageDecoder.decode decodes at once (1536 x 1024 pixels: its activations then fit a 16 GB card beside klein); a larger image is decoded in tiles of IMAGE_TILE tokens overlapping by IMAGE_OVERLAP, blended as H3's decoder's are (fadedTiles).

TEXT_LAYERS

const TEXT_LAYERS = 32

The layers of Qwen3-VL-4B that FLUX.3 reads its text from (loadTextEncoder; convert them with gizml:llm).

H3_TEXT_PICTURE_ROOM

const H3_TEXT_PICTURE_ROOM = 805_306_368

The device bytes the KV pool leaves free for the vision tower's pass over a picture (Vision.seen): 768 MiB, as a 1920 x 1088 picture's pass took 522 MiB on a 16 GB card. Without them the pool took the card and no picture fitted.

ZIMAGE_SHIFT

const ZIMAGE_SHIFT = 3.0

The timestep shift of Z-Image's sampling (ComfyUI's ModelSamplingAuraFlow in its workflow).

ZIMAGE_TEXT_LAYERS

const ZIMAGE_TEXT_LAYERS = 35

ZIMAGE_STEPS

const ZIMAGE_STEPS = 8

The steps Z-Image Turbo is made for (ComfyUI's workflow).