# gizml:ltx

Lightricks' LTX-2.5 (22B), the audio and video flow transformer of giz.ai's ltx2-5-distilled.

Two streams of rows run side by side through 48 blocks: video (4096 wide, 32 heads of 128; a row a latent pixel of 8 frames and 32 x 32 pixels) and audio (2048 wide, 32 heads of 64; a row a 40 ms latent). In each block each stream attends to itself, then to its text, then the audio to the video and the video to the audio, then runs a GELU MLP; each step is modulated (AdaLN) by the stream's timestep, and the text's keys and values by the largest one. Attention reads q and k over their RMS across all heads, then rotated (RoPE by split halves) by the rows' positions (video: time, height, width; audio: time), and its output is gated a head by the input's linear. A text's conditioning (Gemma4's hidden states projected to [rows, 4096 + 2048], the shared encoder's answer) first runs through each stream's connector: 8 blocks of self attention over the rows padded to 1024 with learned registers.

The linears are INT8 rows rotated by ConvRot's Hadamard of 256, a scale a row (ComfyUI's ltx-2.5-22b-distilled-transformer-comfy-int8-convrot file, read as it is: gizml:diffusion's comfyInt8Rows), run on the INT8 matrix cores against x quantized as N.int8Rows quantizes it. Every weight block is in page-locked host memory (gizml:diffusion's Staged) and, while the card has room, on the GPU: the blocks first (each step reads them), then the connectors and the AdaLN modules (a request reads them once: every step's modulation is made at its start). The blocks that stay are spread over the 48, and the others copy to one of two slots, each as soon as its slot is free, so the copies run all through a pass. Ltx.free gives memory back for the VAEs; a block that copies for its use stays on the GPU again while there is room, so the next request gets it back at no cost. On a 16 GB card the copies bound a step, so every byte left on the card counts. The residual streams are F32; every product's output rounds to BF16, as ComfyUI stores it.

```gizlang
import * as Ltx from "gizml:ltx";
```

## Types

### `Dense`

```gizlang
type Dense = { w: Tensor; b: Tensor; }
```

A linear with BF16 weights ([out, in] on the GPU) and its bias ([1, out] F32; [0] for none).

- `w`: `M32.Tensor`
- `b`: `M32.Tensor`

### `Weights`

```gizlang
type Weights = { staged: Ref<Vec<Option<Staged>>>; bytes: Vec<i64>; resident: Ref<Vec<Option<Block>>>; wanted: Vec<bool>; }
```

Blocks in order: each on the GPU while it stays there (resident), in page-locked host memory (staged) once it is
off the GPU (a block resident at load is staged when Ltx.free takes it off, so the host holds only what the card
does not), its staged bytes, and whether it is one to keep there (wanted: the ones that stay are spread over the
blocks, so the copies of the others run while the GPU computes). Each block not resident is staged.

- `staged`: `Ref<Vec<_gizlang_sdk_core.Option<M20.Staged>>>`
- `bytes`: `Vec<i64>`
- `resident`: `Ref<Vec<_gizlang_sdk_core.Option<M20.Block>>>`
- `wanted`: `Vec<bool>`

### `Ltx`

```gizlang
type Ltx = { gpu: Gpu; adaln: Weights; videoIn: Dense; audioIn: Dense; videoOut: Dense; audioOut: Dense; outTables: Vec<Tensor>; marker: Tensor; connectors: Vec<Weights>; registers: Vec<Tensor>; blocks: Weights; slots: Vec<Buffer>; spare: Ref<i64>; least: i64; tables: Vec<Tensor>; ones: Vec<Tensor>; }
```

LTX-2.5 where it runs: its AdaLN modules (8), the input and output projections (video, audio), the output tables
([1, 2, width]), the first frame's marker, the two connectors and their registers, the blocks, the two slots of
the blocks that copy, the bytes a pass needs besides the weights (spare: the request's; least: the load's), and the blocks' tables (T_VIDEO... order,
[blocks, rows, width]); every table F32 of BF16 values on the GPU.

- `gpu`: `M10.Gpu`
- `adaln`: `M24.Weights`
- `videoIn`: `M24.Dense`
- `audioIn`: `M24.Dense`
- `videoOut`: `M24.Dense`
- `audioOut`: `M24.Dense`
- `outTables`: `Vec<M32.Tensor>`
- `marker`: `M32.Tensor`
- `connectors`: `Vec<M24.Weights>`
- `registers`: `Vec<M32.Tensor>`
- `blocks`: `M24.Weights`
- `slots`: `Vec<M10.Buffer>`
- `spare`: `Ref<i64>`
- `least`: `i64`
- `tables`: `Vec<M32.Tensor>`
- `ones`: `Vec<M32.Tensor>`

### `Request`

```gizlang
type Request = { frames: i64; height: i64; width: i64; audio: i64; texts: Vec<Tensor>; videoRope: Vec<Tensor>; audioRope: Vec<Tensor>; videoCross: Vec<Tensor>; audioCross: Vec<Tensor>; guides: Tensor; reference: Tensor; lora: Vec<Lora>; }
```

A request's fixed parts: its latent grid (frames, height, width of the video latent) and audio latents, the two
streams' texts after their connectors ([TEXT_ROWS+, width] BF16), and the rotary tables of each attention (video
self, audio self, the cross attentions' video and audio rows).

- `frames`: `i64`
- `height`: `i64`
- `width`: `i64`
- `audio`: `i64`
- `texts`: `Vec<M32.Tensor>`
- `videoRope`: `Vec<M32.Tensor>`
- `audioRope`: `Vec<M32.Tensor>`
- `videoCross`: `Vec<M32.Tensor>`
- `audioCross`: `Vec<M32.Tensor>`
- `guides`: `M32.Tensor`
- `reference`: `M32.Tensor`
- `lora`: `Vec<M24.Lora>`

### `Guide`

```gizlang
type Guide = { latent: Tensor; frame: i64; scale: i64; }
```

A clean latent that guides a request (LTXVAddGuide at strength 1): its frames ([128, f, H / scale, W / scale] F32,
as VideoEncoder.encode gives them) from pixel frame `frame` on. A `scale` of 2 is an IC-LoRA's small grid
(LTXAddVideoICLoRAGuide's latent_downscale_factor): each row stands for scale x scale latent pixels.

- `latent`: `M32.Tensor`
- `frame`: `i64`
- `scale`: `i64`

### `DurationHead`

```gizlang
type DurationHead = { video: Dense; audio: Dense; videoEmb: Vec<f32>; audioEmb: Vec<f32>; query: Vec<f32>; kv: Tensor; kvBias: Tensor; inW: Vec<f32>; inB: Vec<f32>; outW: Vec<f32>; outB: Vec<f32>; hiddenW: Vec<f32>; hiddenB: Vec<f32>; lastW: Vec<f32>; lastB: Vec<f32>; }
```

giz's autoDuration: LTX's duration head (ComfyUI's DurationHead) over a request's connector rows. Each stream's
rows go to 256 values (plus their stream's embedding); one learned query pools them in 4 heads of 64; an MLP (GELU,
tanh form) gives the log of the seconds.

- `video`: `M24.Dense`
- `audio`: `M24.Dense`
- `videoEmb`: `Vec<f32>`
- `audioEmb`: `Vec<f32>`
- `query`: `Vec<f32>`
- `kv`: `M32.Tensor`
- `kvBias`: `M32.Tensor`
- `inW`: `Vec<f32>`
- `inB`: `Vec<f32>`
- `outW`: `Vec<f32>`
- `outB`: `Vec<f32>`
- `hiddenW`: `Vec<f32>`
- `hiddenB`: `Vec<f32>`
- `lastW`: `Vec<f32>`
- `lastB`: `Vec<f32>`

### `Lora`

```gizlang
type Lora = { a: Vec<Vec<Tensor>>; b: Vec<Vec<Tensor>>; scale: i64; }
```

An IC-LoRA (giz's ingredients, in/outpainting and control workflows): low-rank updates B A, times `strength`, of
the video stream's linears in each block (self attention's q, k, v and out, the text's, the MLP's two). For each
block and linear (V_QKV .. V_DOWN): A [parts r, in] (its parts' A stacked) and B [out, parts r] (their B
block-diagonal, the strength in it), BF16 on the GPU. `scale` is how much smaller its guide latents are
(reference_downscale_factor).

- `a`: `Vec<Vec<M32.Tensor>>`
- `b`: `Vec<Vec<M32.Tensor>>`
- `scale`: `i64`

### `Pass`

```gizlang
type Pass = { video: Vec<Vec<Tensor>>; audio: Vec<Vec<Tensor>>; videoMap: Tensor; audioMap: Tensor; videoRows: Vec<i64>; audioRows: Vec<i64>; videoOut: Vec<Tensor>; audioOut: Vec<Tensor>; }
```

The modulation of one pass where the GPU reads it: for each block, rows of its parameters (Mods' order) at each
time row of its stream, [times, width] F32.

- `video`: `Vec<Vec<M32.Tensor>>`
- `audio`: `Vec<Vec<M32.Tensor>>`
- `videoMap`: `M32.Tensor`
- `audioMap`: `M32.Tensor`
- `videoRows`: `Vec<i64>`
- `audioRows`: `Vec<i64>`
- `videoOut`: `Vec<M32.Tensor>`
- `audioOut`: `Vec<M32.Tensor>`

### `Times`

```gizlang
type Times = { video: Vec<f64>; videoMap: Vec<i64>; audio: Vec<f64>; audioMap: Vec<i64>; }
```

A pass's times: each stream's time rows (sigma in [0, 1]) and the time row of each of its rows.

- `video`: `Vec<f64>`
- `videoMap`: `Vec<i64>`
- `audio`: `Vec<f64>`
- `audioMap`: `Vec<i64>`

### `InPlace`

```gizlang
type InPlace = { latent: Tensor; noise: Tensor; mask: f64; }
```

The first latent frame held in place (ComfyUI's LTXVImgToVideoInplace): its rows `latent` ([k, 128] F32, the
video's first k rows), the start's noise of those rows (unscaled), and the denoise mask 1 - strength.

- `latent`: `M32.Tensor`
- `noise`: `M32.Tensor`
- `mask`: `f64`

### `Conv3`

```gizlang
type Conv3 = { w: Tensor; b: Tensor; }
```

A 3 x 3 x 3 conv: its weight [out, 27 in] BF16 in that order, its bias [out, 1] F32, on the GPU.

- `w`: `M32.Tensor`
- `b`: `M32.Tensor`

### `VaeBlock`

```gizlang
type VaeBlock = { convs: Vec<Conv3>; cell: Vec<i64>; }
```

A decoder block: residual pairs (convs in pairs) or an upsample (one conv, its cell's time, height and width).

- `convs`: `Vec<M24.Conv3>`
- `cell`: `Vec<i64>`

### `VideoVae`

```gizlang
type VideoVae = { gpu: Gpu; input: Conv3; blocks: Vec<VaeBlock>; output: Conv3; mean: Vec<f32>; std: Vec<f32>; }
```

The decoder where it runs: conv_in, the blocks, conv_out, and each latent channel's mean and deviation.

- `gpu`: `M10.Gpu`
- `input`: `M24.Conv3`
- `blocks`: `Vec<M24.VaeBlock>`
- `output`: `M24.Conv3`
- `mean`: `Vec<f32>`
- `std`: `Vec<f32>`

### `Tiles`

```gizlang
type Tiles = { gpu: Gpu; vae: Vec<VideoVae>; fidelity: Vec<DiffusionDecoder>; z: Tensor; ts: Vec<i64>; ys: Vec<i64>; xs: Vec<i64>; wt: Vec<Vec<f32>>; wy: Vec<Vec<f32>>; wx: Vec<Vec<f32>>; next: Ref<i64>; window: Ref<Tensor>; base: Ref<i64>; }
```

A decoding in tiles (VideoVae.tiles): the latent, each side's tile starts and weights, the next temporal tile, and
the output of the last one from its first frame (`base`).

- `gpu`: `M10.Gpu`
- `vae`: `Vec<M24.VideoVae>`
- `fidelity`: `Vec<M24.DiffusionDecoder>`
- `z`: `M32.Tensor`
- `ts`: `Vec<i64>`
- `ys`: `Vec<i64>`
- `xs`: `Vec<i64>`
- `wt`: `Vec<Vec<f32>>`
- `wy`: `Vec<Vec<f32>>`
- `wx`: `Vec<Vec<f32>>`
- `next`: `Ref<i64>`
- `window`: `Ref<M32.Tensor>`
- `base`: `Ref<i64>`

### `NaAttention`

```gizlang
type NaAttention = { kernel: Vec<i64>; qkv: Tensor; bias: Tensor; proj: Dense; qw: Tensor; kw: Tensor; }
```

A neighborhood attention: its window, qkv (q and k rows within each head in split-half order: the file's pairs
2 i, 2 i + 1 become i, i + 32, so gizml:netops' split-half rotary turns them), its bias, the output linear, and
the q and k RMSNorm weights (`[head]` F32, the same order; q's times head^-1/2).

- `kernel`: `Vec<i64>`
- `qkv`: `M32.Tensor`
- `bias`: `M32.Tensor`
- `proj`: `M24.Dense`
- `qw`: `M32.Tensor`
- `kw`: `M32.Tensor`

### `NaBlock`

```gizlang
type NaBlock = { norm1: Tensor; attention: NaAttention; norm2: Tensor; up: Tensor; gate: Tensor; down: Tensor; context: Vec<Dense>; mods: Vec<Tensor>; }
```

A block: its norms (`[c]` F32), attention, SwiGLU (up, gate, down), and in the diffusion stage its context linear and
the step's shift and scale of each norm ([1, c] F32: the AdaLN plus its table).

- `norm1`: `M32.Tensor`
- `attention`: `M24.NaAttention`
- `norm2`: `M32.Tensor`
- `up`: `M32.Tensor`
- `gate`: `M32.Tensor`
- `down`: `M32.Tensor`
- `context`: `Vec<M24.Dense>`
- `mods`: `Vec<M32.Tensor>`

### `NaUpsample`

```gizlang
type NaUpsample = { proj: Dense; stride: Vec<i64>; }
```

A linear pixel shuffle: its linear and stride (time, height, width).

- `proj`: `M24.Dense`
- `stride`: `Vec<i64>`

### `DiffusionDecoder`

```gizlang
type DiffusionDecoder = { gpu: Gpu; input: Dense; stages: Vec<Vec<NaBlock>>; ups: Vec<NaUpsample>; noisy: Dense; blocks: Vec<NaBlock>; norm: Tensor; output: Dense; mean: Vec<f32>; std: Vec<f32>; head: i64; patch: i64; noise: Ref<Tensor>; }
```

The diffusion decoder of the video VAE: its stages, upsamplers, blocks and normalization.

- `gpu`: `M10.Gpu`
- `input`: `M24.Dense`
- `stages`: `Vec<Vec<M24.NaBlock>>`
- `ups`: `Vec<M24.NaUpsample>`
- `noisy`: `M24.Dense`
- `blocks`: `Vec<M24.NaBlock>`
- `norm`: `M32.Tensor`
- `output`: `M24.Dense`
- `mean`: `Vec<f32>`
- `std`: `Vec<f32>`
- `head`: `i64`
- `patch`: `i64`
- `noise`: `Ref<M32.Tensor>`

### `Upsampler`

```gizlang
type Upsampler = { gpu: Gpu; input: Conv3; inNorm: Vec<Tensor>; convs: Vec<Conv3>; norms: Vec<Vec<Tensor>>; up: Tensor; upBias: Tensor; output: Conv3; mean: Vec<f32>; std: Vec<f32>; }
```

giz's quality mode doubles the first stage's latent in space (ComfyUI's LatentUpsampler under LTXVLatentUpsampler):
on the latent without its normalization, a 3D conv to 1024 channels, GroupNorm (32 groups) and SiLU, 4 residual
blocks, a 2D conv to 4 times the channels a frame and a pixel shuffle, 4 residual blocks, a 3D conv back to 128;
then normalized again. Its 3D convs pad with zeros in time as in space.

- `gpu`: `M10.Gpu`
- `input`: `M24.Conv3`
- `inNorm`: `Vec<M32.Tensor>`
- `convs`: `Vec<M24.Conv3>`
- `norms`: `Vec<Vec<M32.Tensor>>`
- `up`: `M32.Tensor`
- `upBias`: `M32.Tensor`
- `output`: `M24.Conv3`
- `mean`: `Vec<f32>`
- `std`: `Vec<f32>`

### `VideoEncoder`

```gizlang
type VideoEncoder = { gpu: Gpu; input: Conv3; blocks: Vec<VaeBlock>; output: Conv3; mean: Vec<f32>; std: Vec<f32>; }
```

The encoder where it runs: conv_in, the blocks (a downsample's cell its stride), conv_out, the latent's statistics.

- `gpu`: `M10.Gpu`
- `input`: `M24.Conv3`
- `blocks`: `Vec<M24.VaeBlock>`
- `output`: `M24.Conv3`
- `mean`: `Vec<f32>`
- `std`: `Vec<f32>`

### `Conv2`

```gizlang
type Conv2 = { w: Tensor; b: Tensor; size: i64; }
```

A 2D conv: its weight [out, in k k] F32 and bias [out, 1] F32, on the GPU.

- `w`: `M32.Tensor`
- `b`: `M32.Tensor`
- `size`: `i64`

### `AudioRes`

```gizlang
type AudioRes = { convs: Vec<Conv2>; }
```

A residual block: conv1, conv2 and the 1 x 1 shortcut where the channels change.

- `convs`: `Vec<M24.Conv2>`

### `AudioVae`

```gizlang
type AudioVae = { gpu: Gpu; input: Conv2; blocks: Vec<AudioRes>; ups: Vec<i64>; upConvs: Vec<Conv2>; output: Conv2; mean: Vec<f32>; std: Vec<f32>; vocoder: D.BigVgan; bwe: D.BigVgan; stft: Tensor; mel: Tensor; }
```

The audio decoder: conv_in, the residual blocks in order, after which block (index) each upsample (its conv)
comes, conv_out, the latent statistics ([128] each), the two vocoders and the extension's STFT and mel bases
([514, 512] and [64, 257] F32 on the GPU).

- `gpu`: `M10.Gpu`
- `input`: `M24.Conv2`
- `blocks`: `Vec<M24.AudioRes>`
- `ups`: `Vec<i64>`
- `upConvs`: `Vec<M24.Conv2>`
- `output`: `M24.Conv2`
- `mean`: `Vec<f32>`
- `std`: `Vec<f32>`
- `vocoder`: `M20.BigVgan`
- `bwe`: `M20.BigVgan`
- `stft`: `M32.Tensor`
- `mel`: `M32.Tensor`

### `AudioEncoder`

```gizlang
type AudioEncoder = { gpu: Gpu; input: Conv2; blocks: Vec<AudioRes>; downs: Vec<i64>; downConvs: Vec<Conv2>; output: Conv2; mean: Vec<f32>; std: Vec<f32>; basis: Tensor; mel: Tensor; }
```

The audio encoder: conv_in, the residual blocks in order, after which block each downsample (its conv) comes,
conv_out, the latent statistics, and the STFT and mel bases ([1026, 1024] and [64, 513] F32 on the GPU).

- `gpu`: `M10.Gpu`
- `input`: `M24.Conv2`
- `blocks`: `Vec<M24.AudioRes>`
- `downs`: `Vec<i64>`
- `downConvs`: `Vec<M24.Conv2>`
- `output`: `M24.Conv2`
- `mean`: `Vec<f32>`
- `std`: `Vec<f32>`
- `basis`: `M32.Tensor`
- `mel`: `M32.Tensor`

### `Clip`

```gizlang
type Clip = { c: i64; h: i64; w: i64; frames: Vec<bytes>; }
```

n frames of c planes of h x w bytes (RGB: c = 3; a mask: c = 1, 255 is 1).

- `c`: `i64`
- `h`: `i64`
- `w`: `i64`
- `frames`: `Vec<bytes>`

## Functions

### `load`

```gizlang
function load(sys: System, path: string, gpu: Gpu, spare: i64): Ltx throws string
```

LTX-2.5 from ComfyUI's INT8 file `path` on `gpu`: its weights stay on the GPU (the blocks first, then the
connectors and the AdaLN modules) while it keeps `spare` bytes free besides the slots, for a pass's rows (measured:
0.7 GB at 832 x 480 x 97).

Effects: fail, read, write, console, io.

### `audioLatents`

```gizlang
function audioLatents(frames: i64, rate: f64): i64
```

The audio latents of a video of `frames` frames at `rate` frames a second, as LTXVEmptyLatentAudio counts them:
its seconds times 25 latents a second (16 kHz, hop 160, 4 mel frames a latent), rounded half to even.

Effects: fail.

### `loadDurationHead`

```gizlang
function loadDurationHead(sys: System, path: string, gpu: Gpu): DurationHead throws string
```

The duration head of the checkpoint at PATH on GPU.

Effects: fail, read, write, console, io.

### `durationFrames`

```gizlang
function durationFrames(seconds: f64, rate: f64, least: f64, most: f64): i64
```

`seconds` at `rate` frames a second as frames 1 + 8 k: clamped to `least` .. `most` seconds, then rounded down
to the grid (up, when that falls short of the least).

Effects: fail.

### `loadLora`

```gizlang
function loadLora(sys: System, path: string, strength: f64, gpu: Gpu): Lora throws string
```

The LoRA in file `path` at `strength` on `gpu`.

Effects: fail, read, write, console, io.

### `at`

```gizlang
function at(sigma: f64, video: i64, audio: i64): Times
```

Times of a pass with every row of `video` video rows and `audio` audio rows at `sigma`.

Effects: fail.

### `sharedText`

```gizlang
function sharedText(sys: System, host: string, port: i64, prompt: string, authorization: string): Tensor throws string
```

The conditioning of `prompt` ([rows, 6144] F32 on the host, as Ltx.request takes it) from giz's shared LTX-2.5
text encoder at host:port (POST /api/ltx25/text-embeddings with the prompt and an empty negative prompt, and
`authorization` when not empty; it answers with a ZIP of the two conditionings' safetensors). The request's bytes
are those of its other clients, so its cache serves both.

Effects: fail, read, write, console, io.

### `distilledSigmas`

```gizlang
function distilledSigmas(): Vec<f64>
```

The noise levels of the distilled model's steps, from 1 to 0.

Effects: fail.

### `noise`

```gizlang
function noise(seed: u64, frames: i64, height: i64, width: i64, audio: i64): Vec<Tensor>
```

Seed `seed`'s noise as ComfyUI's RandomNoise draws it for an audio and video latent (torch.randn of the video
latent [1, 128, frames, height, width], then of the audio latent [1, 8, audio, 16], on one CPU generator), as rows:
[video rows, audio rows].

Effects: fail, read, write, console, io.

### `guidedNoise`

```gizlang
function guidedNoise(seed: u64, frames: i64, height: i64, width: i64, audio: i64, guides: Vec<Guide>): Vec<Tensor>
```

noise() for a request with `guides`: drawn over the video's frames and the guides' at the full grid (as ComfyUI
draws it over the latent with the guides appended), the video's rows kept and the guides' rows zero (each step sets
them).

Effects: fail, read, write, console, io.

### `videoLatent`

```gizlang
function videoLatent(rows: Tensor, frames: i64, height: i64, width: i64): Tensor
```

A video latent's rows ([pixels, 128]) as the latent [128, frames, height, width] F32 on the host.

Effects: fail, read, write, console, io.

### `audioLatent`

```gizlang
function audioLatent(rows: Tensor, audio: i64): Tensor
```

An audio latent's rows ([audio, 128]) as the latent [8, audio, 16].

Effects: fail, read, write, console, io.

### `loadVideoVae`

```gizlang
function loadVideoVae(sys: System, path: string, gpu: Gpu): VideoVae throws string
```

The video decoder in ComfyUI's file `path` on `gpu`.

Effects: fail, read, write, console, io.

### `loadDiffusionDecoder`

```gizlang
function loadDiffusionDecoder(sys: System, path: string, gpu: Gpu): DiffusionDecoder throws string
```

The fidelity decoder in ComfyUI's file `path` on `gpu`.

Effects: fail, read, write, console, io.

### `loadUpsampler`

```gizlang
function loadUpsampler(sys: System, path: string, vae: string, gpu: Gpu): Upsampler throws string
```

The upsampler in ComfyUI's file `path` on `gpu`, with the latent statistics of the video VAE at `vae`.

Effects: fail, read, write, console, io.

### `loadVideoEncoder`

```gizlang
function loadVideoEncoder(sys: System, path: string, gpu: Gpu): VideoEncoder throws string
```

The encoder in ComfyUI's file `path` on `gpu`.

Effects: fail, read, write, console, io.

### `guideFrames`

```gizlang
function guideFrames(images: Vec<Image.Image>, width: i64, height: i64): Tensor
```

`images` as an encoder's frames of width x height pixels ([n, 3, height, width] F32 in [-1, 1]), each cut to the
output's aspect about its center and resized by bilinear, as ComfyUI's LTXVAddGuide makes a guide's pixels.

Effects: fail.

### `loadAudioVae`

```gizlang
function loadAudioVae(sys: System, path: string, gpu: Gpu): AudioVae throws string
```

LTX-2.5's audio decoder and vocoders in ComfyUI's file `path` on `gpu`.

Effects: fail, read, write, console, io.

### `loadAudioEncoder`

```gizlang
function loadAudioEncoder(sys: System, path: string, gpu: Gpu): AudioEncoder throws string
```

The audio encoder in ComfyUI's audio VAE file `path` on `gpu`.

Effects: fail, read, write, console, io.

### `clipOf`

```gizlang
function clipOf(images: Vec<Image.Image>): Clip
```

The frames of RGB images (interleaved bytes) as a clip.

Effects: fail.

### `clipOfBytes`

```gizlang
function clipOfBytes(c: i64, h: i64, w: i64, frames: Vec<bytes>): Clip
```

A clip of frames of c planes of h x w bytes each.

### `clipOfRgb`

```gizlang
function clipOfRgb(h: i64, w: i64, frames: Vec<bytes>): Clip
```

A clip of interleaved RGB frames (ffmpeg's rgb24) of h x w.

Effects: fail.

### `blendedFrame`

```gizlang
function blendedFrame(a: Vec<f32>, b: bytes, mask: bytes, h: i64, w: i64, dilation: i64): Vec<f32>
```

ComfyUI's LTXVLaplacianPyramidBlend of frame a (generated, F32 [3, h, w] in [0, 1]) over frame b (bytes) by `mask`
(one plane of bytes): the mask widened at a long side of 64 by `dilation` (bilinear down, max, bilinear up), the
planes padded to powers of two, Laplacian pyramids of up to 7 levels mixed by the mask's Gaussian pyramid, cut back
and clamped to [0, 1].

Effects: fail.

### `blended`

```gizlang
function blended(a: Tensor, b: Clip, mask: Clip, at: i64, dilation: i64): Tensor
```

blendedFrame of each frame of `a` ([k, 3, h, w] F32 in [0, 1] on a GPU) over frames at .. at + k of `b` by `mask`:
the same on the GPU.

Effects: fail, read, write, console, io.

## Methods

### `Ltx.request`

```gizlang
function Ltx.request(self: Ltx, texts: Vec<Tensor>, frames: i64, height: i64, width: i64, audio0: i64, rate: f64, guides: Vec<Guide>, reference: Tensor, lora: Vec<Lora>): Request throws string
```

A request for a video latent of `frames` x `height` x `width` (latent pixels) and `audio` audio latents at `rate`
frames a second, conditioned on `texts` (Ltx.connect) and guided by `guides` (their frames' rows follow the
video's). A soundtrack `reference` (its latent rows, AudioEncoder.encode; or none, [0, 128]) is the clip's audio,
kept as it is, and the model also reads it as a voice before the clip (ComfyUI's LTXVSetAudioRefTokens).

Effects: fail, read, write, console, io.

### `DurationHead.seconds`

```gizlang
function DurationHead.seconds(self: DurationHead, texts: Vec<Tensor>): f64
```

The seconds the head predicts for connector rows `texts` (Ltx.connect).

Effects: fail, read, write, console, io.

### `Ltx.connect`

```gizlang
function Ltx.connect(self: Ltx, text: Tensor): Vec<Tensor> throws string
```

The text `text` ([rows, 6144]: video's 4096 values, then audio's 2048, as the shared encoder answers) through
both connectors: [video rows, audio rows] as Ltx.request and the duration head take them.

Effects: fail, read, write, console, io.

### `Ltx.free`

```gizlang
function Ltx.free(self: Ltx, bytes: i64): void
```

Gives the GPU's memory back until `bytes` more fit: the AdaLN modules, the connectors, then the last blocks leave
it, each staged first if it was not (the copy stays; a block's next use puts it back while there is room).

Effects: fail, read, write, io.

### `Ltx.resident`

```gizlang
function Ltx.resident(self: Ltx): i64
```

The blocks on the GPU now, of 48.

Effects: fail, read.

### `Ltx.passes`

```gizlang
function Ltx.passes(self: Ltx, list: Vec<Times>): Vec<Pass>
```

The modulation of passes at `list`, the AdaLN modules run once over all their time rows.

Effects: fail, read, write, console, io.

### `Ltx.velocity`

```gizlang
function Ltx.velocity(self: Ltx, r: Request, video: Tensor, audio: Tensor, p: Pass): Vec<Tensor>
```

The velocity of LTX-2.5 at video rows `video` ([rows, 128] F32 on the GPU: the latent's pixels in (frame, height,
width) order) and audio rows `audio` ([rows, 128]: channel, then bin), modulated by pass p (Ltx.passes) of their
times. [video velocity, audio velocity] as F32 rows.

Effects: fail, read, write, console, io.

### `Ltx.sample`

```gizlang
function Ltx.sample(self: Ltx, r: Request, start: Vec<Tensor>, sigmas: Vec<f64>, ancestral: bool, seed: u64, progress: (step: i64, total: i64) => bool, inPlace: Vec<InPlace> = []): Vec<Tensor>
```

Samples a request from noise rows (noise()) over `sigmas` with Euler steps, ancestral (eta 1: each step goes to
sigma_next² / sigma, then renoises to sigma_next, as ComfyUI's euler_ancestral for a flow model) when `ancestral`,
the renoising drawn from `seed` + 1 on the CPU generator, as ComfyUI does on a CPU (on a GPU it draws CUDA's, so its
steps after the first differ from ours in their noise only). Every row is at the step's sigma but the guides' rows
(the noise's last video rows): they stay the guides' clean latent at time 0, as ComfyUI's denoise mask keeps them.
`progress` is told each step before it runs (its index and the steps): false stops there. With `inPlace`, its rows
are at time sigma times its mask, the model sees them mixed with their noisy latent and its denoised rows are mixed
with the latent by the mask (ComfyUI's KSamplerX0Inpaint). [video rows, audio rows] F32 on the GPU, the guides' rows
left out.

Effects: fail, read, write, console, io.

### `VideoVae.decode`

```gizlang
function VideoVae.decode(self: VideoVae, latent: Tensor): Tensor
```

The frames of `latent` ([128, F, H, W] F32): [8 F - 7, 3, 32 H, 32 W] F32 on the GPU, in [-1, 1] (not clamped).

Effects: fail, read, write, console, io.

### `VideoVae.tiles`

```gizlang
function VideoVae.tiles(self: VideoVae, latent: Tensor): Tiles
```

The decoding of `latent` ([128, F, H, W] F32 on the host) in tiles: Tiles.next gives its frames in order.

Effects: fail, read, write, console, io.

### `DiffusionDecoder.tiles`

```gizlang
function DiffusionDecoder.tiles(self: DiffusionDecoder, latent: Tensor): Tiles
```

The fidelity decoder's tiles of `latent`, as VideoVae.tiles.

Effects: fail, read, write, console, io.

### `Tiles.next`

```gizlang
function Tiles.next(self: Tiles): Option<Tensor>
```

The next frames that are final, [k, 3, 32 H, 32 W] F32 on the GPU in [0, 1]: those of the next temporal tile
before the one after it starts; none when all are given.

Effects: fail, read, write, console, io.

### `DiffusionDecoder.decode`

```gizlang
function DiffusionDecoder.decode(self: DiffusionDecoder, latent: Tensor): Tensor
```

The frames of `latent` ([128, F, H, W] normalized, on the host): [8 F - 7, 3, 32 H, 32 W] F32 on the GPU in
[-1, 1].

Effects: fail, read, write, console, io.

### `Upsampler.run`

```gizlang
function Upsampler.run(self: Upsampler, latent: Tensor): Tensor
```

`latent` ([128, frames, h, w], normalized as the VAE's latents are) at twice its size.

Effects: fail, read, write, console, io.

### `VideoEncoder.encode`

```gizlang
function VideoEncoder.encode(self: VideoEncoder, frames: Tensor): Tensor
```

The latent ([128, k + 1, H / 32, W / 32] F32 on the host) of `frames` ([8 k + 1, 3, H, W] F32 in [-1, 1]).

Effects: fail, read, write, console, io.

### `VideoEncoder.encodeFrom`

```gizlang
function VideoEncoder.encodeFrom(self: VideoEncoder, n: i64, height: i64, width: i64, frames: (at: i64, m: i64) => Tensor): Tensor
```

encode() of n frames of H x W that `frames` gives a run at a time: frames at .. at + m ([m, 3, H, W] F32 in
[-1, 1]), so the whole clip is never in memory.

Effects: fail, read, write, console, io.

### `AudioVae.decode`

```gizlang
function AudioVae.decode(self: AudioVae, rows: Tensor): Tensor
```

The 48 kHz stereo audio of audio latent rows ([T, 128]): [2, samples] F32 on the host, in [-1, 1].

Effects: fail, read, write, console, io.

### `AudioEncoder.encode`

```gizlang
function AudioEncoder.encode(self: AudioEncoder, left: Vec<f32>, right: Vec<f32>): Tensor
```

Stereo 16 kHz `left` and `right` as normalized latent rows [frames / 4, 128] on the GPU (channel, then bin).

Effects: fail, read, write, console, io.

### `Clip.images`

```gizlang
function Clip.images(self: Clip): Vec<Image.Image>
```

The frames as RGB images (interleaved bytes).

Effects: fail.

### `Clip.red`

```gizlang
function Clip.red(self: Clip): Clip
```

One plane of the clip's red values: a mask video as ComfyUI's ImageToMask (channel red) reads it.

Effects: fail.

### `Clip.scaled`

```gizlang
function Clip.scaled(self: Clip, oh: i64, ow: i64, mode: string, crop: bool): Clip
```

The clip resized to oh x ow by mode ("lanczos", "area", "bilinear", "nearest"), its center cut first to the new
aspect when `crop` (ComfyUI's common_upscale).

Effects: fail.

### `Clip.shorter`

```gizlang
function Clip.shorter(self: Clip, side: i64, mode: string): Clip
```

ComfyUI's "scale shorter dimension" to `side` (the other side rounded half to even).

Effects: fail.

### `Clip.toMultiple`

```gizlang
function Clip.toMultiple(self: Clip, k: i64, mode: string): Clip
```

ComfyUI's "scale to multiple": the largest multiple of `k` each side, covered by a resize, then cut at the
center.

Effects: fail.

### `Clip.padded`

```gizlang
function Clip.padded(self: Clip, width: i64, height: i64): Vec<Clip>
```

ComfyUI's ImagePadForOutpaintTargetSize (nearest-exact, no feathering): the frames shrunk to fit width x height if
larger, centered on gray 0.5; and the mask, 1 in the padding. [frames, mask].

Effects: fail.

### `Clip.green`

```gizlang
function Clip.green(self: Clip, mask: Clip): Clip
```

The frames painted green where `mask` is set (LTXVInpaintPreprocess), the shorter of the two clips long.

Effects: fail.

### `Clip.dilated`

```gizlang
function Clip.dilated(self: Clip, radius: i64): Clip
```

A mask's frames dilated by `radius` pixels (max over the square around each) and cut at 0.5 (LTXVDilateVideoMask
without time).

Effects: fail.

### `Clip.pixels`

```gizlang
function Clip.pixels(self: Clip, at: i64, m: i64): Tensor
```

Frames at .. at + m as [m, 3, h, w] F32 in [-1, 1] (VideoEncoder.encodeFrom).

Effects: fail.

### `Clip.on`

```gizlang
function Clip.on(self: Clip, gpu: Gpu, at: i64, m: i64): Tensor
```

The frames as F32 [n, 3, h, w] values in [0, 1] on `gpu`, frames at .. at + m.

Effects: fail, read, write, console, io.

### `Clip.edges`

```gizlang
function Clip.edges(self: Clip, low: f64, high: f64): Clip
```

Each frame's Canny edges as OpenCV's Canny (3 x 3 Sobel with the edge repeated, the largest gradient of the three
colors, L1 magnitudes, thinned along the gradient, kept above `high` or above `low` when joined to one), white on
black in all three planes: ComfyUI's CannyEdgePreprocessor at the frames' size.

Effects: fail.

## Constants

### `TOO_LARGE`

```gizlang
const TOO_LARGE = "too large for this GPU"
```

What a request asks that does not fit this GPU (Ltx.request).

### `VIDEO`

```gizlang
const VIDEO = 4096
```

The width of a video row.

### `AUDIO`

```gizlang
const AUDIO = 2048
```

The width of an audio row.

### `LATENT`

```gizlang
const LATENT = 128
```

The channels of a video latent.

### `AUDIO_CHANNELS`

```gizlang
const AUDIO_CHANNELS = 8
```

The channels of an audio latent (a row is its channels times its frequency bins, 128 values).

### `AUDIO_BINS`

```gizlang
const AUDIO_BINS = 16
```

The frequency bins of an audio latent.

### `FRAMES_A_LATENT`

```gizlang
const FRAMES_A_LATENT = 8
```

The video frames of one latent frame of the video VAE.

### `PIXELS_A_LATENT`

```gizlang
const PIXELS_A_LATENT = 32
```

The pixels of one latent pixel of the video VAE, in height and in width.

gizml:ltx

Lightricks' LTX-2.5 (22B), the audio and video flow transformer of giz.ai's ltx2-5-distilled.

Two streams of rows run side by side through 48 blocks: video (4096 wide, 32 heads of 128; a row a latent pixel of 8 frames and 32 x 32 pixels) and audio (2048 wide, 32 heads of 64; a row a 40 ms latent). In each block each stream attends to itself, then to its text, then the audio to the video and the video to the audio, then runs a GELU MLP; each step is modulated (AdaLN) by the stream's timestep, and the text's keys and values by the largest one. Attention reads q and k over their RMS across all heads, then rotated (RoPE by split halves) by the rows' positions (video: time, height, width; audio: time), and its output is gated a head by the input's linear. A text's conditioning (Gemma4's hidden states projected to [rows, 4096 + 2048], the shared encoder's answer) first runs through each stream's connector: 8 blocks of self attention over the rows padded to 1024 with learned registers.

The linears are INT8 rows rotated by ConvRot's Hadamard of 256, a scale a row (ComfyUI's ltx-2.5-22b-distilled-transformer-comfy-int8-convrot file, read as it is: gizml:diffusion's comfyInt8Rows), run on the INT8 matrix cores against x quantized as N.int8Rows quantizes it. Every weight block is in page-locked host memory (gizml:diffusion's Staged) and, while the card has room, on the GPU: the blocks first (each step reads them), then the connectors and the AdaLN modules (a request reads them once: every step's modulation is made at its start). The blocks that stay are spread over the 48, and the others copy to one of two slots, each as soon as its slot is free, so the copies run all through a pass. Ltx.free gives memory back for the VAEs; a block that copies for its use stays on the GPU again while there is room, so the next request gets it back at no cost. On a 16 GB card the copies bound a step, so every byte left on the card counts. The residual streams are F32; every product's output rounds to BF16, as ComfyUI stores it.

gizlang

import * as Ltx from "gizml:ltx";

Types

`Dense`

gizlang

type Dense = { w: Tensor; b: Tensor; }

A linear with BF16 weights ([out, in] on the GPU) and its bias ([1, out] F32; [0] for none).

w: M32.Tensor
b: M32.Tensor

`Weights`

gizlang

type Weights = { staged: Ref<Vec<Option<Staged>>>; bytes: Vec<i64>; resident: Ref<Vec<Option<Block>>>; wanted: Vec<bool>; }

Blocks in order: each on the GPU while it stays there (resident), in page-locked host memory (staged) once it is off the GPU (a block resident at load is staged when Ltx.free takes it off, so the host holds only what the card does not), its staged bytes, and whether it is one to keep there (wanted: the ones that stay are spread over the blocks, so the copies of the others run while the GPU computes). Each block not resident is staged.

staged: Ref<Vec<_gizlang_sdk_core.Option<M20.Staged>>>
bytes: Vec<i64>
resident: Ref<Vec<_gizlang_sdk_core.Option<M20.Block>>>
wanted: Vec<bool>

`Ltx`

gizlang

type Ltx = { gpu: Gpu; adaln: Weights; videoIn: Dense; audioIn: Dense; videoOut: Dense; audioOut: Dense; outTables: Vec<Tensor>; marker: Tensor; connectors: Vec<Weights>; registers: Vec<Tensor>; blocks: Weights; slots: Vec<Buffer>; spare: Ref<i64>; least: i64; tables: Vec<Tensor>; ones: Vec<Tensor>; }

LTX-2.5 where it runs: its AdaLN modules (8), the input and output projections (video, audio), the output tables ([1, 2, width]), the first frame's marker, the two connectors and their registers, the blocks, the two slots of the blocks that copy, the bytes a pass needs besides the weights (spare: the request's; least: the load's), and the blocks' tables (T_VIDEO... order, [blocks, rows, width]); every table F32 of BF16 values on the GPU.

gpu: M10.Gpu
adaln: M24.Weights
videoIn: M24.Dense
audioIn: M24.Dense
videoOut: M24.Dense
audioOut: M24.Dense
outTables: Vec<M32.Tensor>
marker: M32.Tensor
connectors: Vec<M24.Weights>
registers: Vec<M32.Tensor>
blocks: M24.Weights
slots: Vec<M10.Buffer>
spare: Ref<i64>
least: i64
tables: Vec<M32.Tensor>
ones: Vec<M32.Tensor>

`Request`

gizlang

type Request = { frames: i64; height: i64; width: i64; audio: i64; texts: Vec<Tensor>; videoRope: Vec<Tensor>; audioRope: Vec<Tensor>; videoCross: Vec<Tensor>; audioCross: Vec<Tensor>; guides: Tensor; reference: Tensor; lora: Vec<Lora>; }

A request's fixed parts: its latent grid (frames, height, width of the video latent) and audio latents, the two streams' texts after their connectors ([TEXT_ROWS+, width] BF16), and the rotary tables of each attention (video self, audio self, the cross attentions' video and audio rows).

frames: i64
height: i64
width: i64
audio: i64
texts: Vec<M32.Tensor>
videoRope: Vec<M32.Tensor>
audioRope: Vec<M32.Tensor>
videoCross: Vec<M32.Tensor>
audioCross: Vec<M32.Tensor>
guides: M32.Tensor
reference: M32.Tensor
lora: Vec<M24.Lora>

`Guide`

gizlang

type Guide = { latent: Tensor; frame: i64; scale: i64; }

A clean latent that guides a request (LTXVAddGuide at strength 1): its frames ([128, f, H / scale, W / scale] F32, as VideoEncoder.encode gives them) from pixel frame frame on. A scale of 2 is an IC-LoRA's small grid (LTXAddVideoICLoRAGuide's latent_downscale_factor): each row stands for scale x scale latent pixels.

latent: M32.Tensor
frame: i64
scale: i64

`DurationHead`

gizlang

type DurationHead = { video: Dense; audio: Dense; videoEmb: Vec<f32>; audioEmb: Vec<f32>; query: Vec<f32>; kv: Tensor; kvBias: Tensor; inW: Vec<f32>; inB: Vec<f32>; outW: Vec<f32>; outB: Vec<f32>; hiddenW: Vec<f32>; hiddenB: Vec<f32>; lastW: Vec<f32>; lastB: Vec<f32>; }

giz's autoDuration: LTX's duration head (ComfyUI's DurationHead) over a request's connector rows. Each stream's rows go to 256 values (plus their stream's embedding); one learned query pools them in 4 heads of 64; an MLP (GELU, tanh form) gives the log of the seconds.

video: M24.Dense
audio: M24.Dense
videoEmb: Vec<f32>
audioEmb: Vec<f32>
query: Vec<f32>
kv: M32.Tensor
kvBias: M32.Tensor
inW: Vec<f32>
inB: Vec<f32>
outW: Vec<f32>
outB: Vec<f32>
hiddenW: Vec<f32>
hiddenB: Vec<f32>
lastW: Vec<f32>
lastB: Vec<f32>

`Lora`

gizlang

type Lora = { a: Vec<Vec<Tensor>>; b: Vec<Vec<Tensor>>; scale: i64; }

An IC-LoRA (giz's ingredients, in/outpainting and control workflows): low-rank updates B A, times strength, of the video stream's linears in each block (self attention's q, k, v and out, the text's, the MLP's two). For each block and linear (V_QKV .. V_DOWN): A [parts r, in] (its parts' A stacked) and B [out, parts r] (their B block-diagonal, the strength in it), BF16 on the GPU. scale is how much smaller its guide latents are (reference_downscale_factor).

a: Vec<Vec<M32.Tensor>>
b: Vec<Vec<M32.Tensor>>
scale: i64

`Pass`

gizlang

type Pass = { video: Vec<Vec<Tensor>>; audio: Vec<Vec<Tensor>>; videoMap: Tensor; audioMap: Tensor; videoRows: Vec<i64>; audioRows: Vec<i64>; videoOut: Vec<Tensor>; audioOut: Vec<Tensor>; }

The modulation of one pass where the GPU reads it: for each block, rows of its parameters (Mods' order) at each time row of its stream, [times, width] F32.

video: Vec<Vec<M32.Tensor>>
audio: Vec<Vec<M32.Tensor>>
videoMap: M32.Tensor
audioMap: M32.Tensor
videoRows: Vec<i64>
audioRows: Vec<i64>
videoOut: Vec<M32.Tensor>
audioOut: Vec<M32.Tensor>

`Times`

gizlang

type Times = { video: Vec<f64>; videoMap: Vec<i64>; audio: Vec<f64>; audioMap: Vec<i64>; }

A pass's times: each stream's time rows (sigma in [0, 1]) and the time row of each of its rows.

video: Vec<f64>
videoMap: Vec<i64>
audio: Vec<f64>
audioMap: Vec<i64>

`InPlace`

gizlang

type InPlace = { latent: Tensor; noise: Tensor; mask: f64; }

The first latent frame held in place (ComfyUI's LTXVImgToVideoInplace): its rows latent ([k, 128] F32, the video's first k rows), the start's noise of those rows (unscaled), and the denoise mask 1 - strength.

latent: M32.Tensor
noise: M32.Tensor
mask: f64

`Conv3`

gizlang

type Conv3 = { w: Tensor; b: Tensor; }

A 3 x 3 x 3 conv: its weight [out, 27 in] BF16 in that order, its bias [out, 1] F32, on the GPU.

w: M32.Tensor
b: M32.Tensor

`VaeBlock`

gizlang

type VaeBlock = { convs: Vec<Conv3>; cell: Vec<i64>; }

A decoder block: residual pairs (convs in pairs) or an upsample (one conv, its cell's time, height and width).

convs: Vec<M24.Conv3>
cell: Vec<i64>

`VideoVae`

gizlang

type VideoVae = { gpu: Gpu; input: Conv3; blocks: Vec<VaeBlock>; output: Conv3; mean: Vec<f32>; std: Vec<f32>; }

The decoder where it runs: conv_in, the blocks, conv_out, and each latent channel's mean and deviation.

gpu: M10.Gpu
input: M24.Conv3
blocks: Vec<M24.VaeBlock>
output: M24.Conv3
mean: Vec<f32>
std: Vec<f32>

`Tiles`

gizlang

type Tiles = { gpu: Gpu; vae: Vec<VideoVae>; fidelity: Vec<DiffusionDecoder>; z: Tensor; ts: Vec<i64>; ys: Vec<i64>; xs: Vec<i64>; wt: Vec<Vec<f32>>; wy: Vec<Vec<f32>>; wx: Vec<Vec<f32>>; next: Ref<i64>; window: Ref<Tensor>; base: Ref<i64>; }

A decoding in tiles (VideoVae.tiles): the latent, each side's tile starts and weights, the next temporal tile, and the output of the last one from its first frame (base).

gpu: M10.Gpu
vae: Vec<M24.VideoVae>
fidelity: Vec<M24.DiffusionDecoder>
z: M32.Tensor
ts: Vec<i64>
ys: Vec<i64>
xs: Vec<i64>
wt: Vec<Vec<f32>>
wy: Vec<Vec<f32>>
wx: Vec<Vec<f32>>
next: Ref<i64>
window: Ref<M32.Tensor>
base: Ref<i64>

`NaAttention`

gizlang

type NaAttention = { kernel: Vec<i64>; qkv: Tensor; bias: Tensor; proj: Dense; qw: Tensor; kw: Tensor; }

A neighborhood attention: its window, qkv (q and k rows within each head in split-half order: the file's pairs 2 i, 2 i + 1 become i, i + 32, so gizml:netops' split-half rotary turns them), its bias, the output linear, and the q and k RMSNorm weights ([head] F32, the same order; q's times head^-1/2).

kernel: Vec<i64>
qkv: M32.Tensor
bias: M32.Tensor
proj: M24.Dense
qw: M32.Tensor
kw: M32.Tensor

`NaBlock`

gizlang

type NaBlock = { norm1: Tensor; attention: NaAttention; norm2: Tensor; up: Tensor; gate: Tensor; down: Tensor; context: Vec<Dense>; mods: Vec<Tensor>; }

A block: its norms ([c] F32), attention, SwiGLU (up, gate, down), and in the diffusion stage its context linear and the step's shift and scale of each norm ([1, c] F32: the AdaLN plus its table).

norm1: M32.Tensor
attention: M24.NaAttention
norm2: M32.Tensor
up: M32.Tensor
gate: M32.Tensor
down: M32.Tensor
context: Vec<M24.Dense>
mods: Vec<M32.Tensor>

`NaUpsample`

gizlang

type NaUpsample = { proj: Dense; stride: Vec<i64>; }

A linear pixel shuffle: its linear and stride (time, height, width).

proj: M24.Dense
stride: Vec<i64>

`DiffusionDecoder`

gizlang

type DiffusionDecoder = { gpu: Gpu; input: Dense; stages: Vec<Vec<NaBlock>>; ups: Vec<NaUpsample>; noisy: Dense; blocks: Vec<NaBlock>; norm: Tensor; output: Dense; mean: Vec<f32>; std: Vec<f32>; head: i64; patch: i64; noise: Ref<Tensor>; }

The diffusion decoder of the video VAE: its stages, upsamplers, blocks and normalization.

gpu: M10.Gpu
input: M24.Dense
stages: Vec<Vec<M24.NaBlock>>
ups: Vec<M24.NaUpsample>
noisy: M24.Dense
blocks: Vec<M24.NaBlock>
norm: M32.Tensor
output: M24.Dense
mean: Vec<f32>
std: Vec<f32>
head: i64
patch: i64
noise: Ref<M32.Tensor>

`Upsampler`

gizlang

type Upsampler = { gpu: Gpu; input: Conv3; inNorm: Vec<Tensor>; convs: Vec<Conv3>; norms: Vec<Vec<Tensor>>; up: Tensor; upBias: Tensor; output: Conv3; mean: Vec<f32>; std: Vec<f32>; }

giz's quality mode doubles the first stage's latent in space (ComfyUI's LatentUpsampler under LTXVLatentUpsampler): on the latent without its normalization, a 3D conv to 1024 channels, GroupNorm (32 groups) and SiLU, 4 residual blocks, a 2D conv to 4 times the channels a frame and a pixel shuffle, 4 residual blocks, a 3D conv back to 128; then normalized again. Its 3D convs pad with zeros in time as in space.

gpu: M10.Gpu
input: M24.Conv3
inNorm: Vec<M32.Tensor>
convs: Vec<M24.Conv3>
norms: Vec<Vec<M32.Tensor>>
up: M32.Tensor
upBias: M32.Tensor
output: M24.Conv3
mean: Vec<f32>
std: Vec<f32>

`VideoEncoder`

gizlang

type VideoEncoder = { gpu: Gpu; input: Conv3; blocks: Vec<VaeBlock>; output: Conv3; mean: Vec<f32>; std: Vec<f32>; }

The encoder where it runs: conv_in, the blocks (a downsample's cell its stride), conv_out, the latent's statistics.

gpu: M10.Gpu
input: M24.Conv3
blocks: Vec<M24.VaeBlock>
output: M24.Conv3
mean: Vec<f32>
std: Vec<f32>

`Conv2`

gizlang

type Conv2 = { w: Tensor; b: Tensor; size: i64; }

A 2D conv: its weight [out, in k k] F32 and bias [out, 1] F32, on the GPU.

w: M32.Tensor
b: M32.Tensor
size: i64

`AudioRes`

gizlang

type AudioRes = { convs: Vec<Conv2>; }

A residual block: conv1, conv2 and the 1 x 1 shortcut where the channels change.

convs: Vec<M24.Conv2>

`AudioVae`

gizlang

type AudioVae = { gpu: Gpu; input: Conv2; blocks: Vec<AudioRes>; ups: Vec<i64>; upConvs: Vec<Conv2>; output: Conv2; mean: Vec<f32>; std: Vec<f32>; vocoder: D.BigVgan; bwe: D.BigVgan; stft: Tensor; mel: Tensor; }

The audio decoder: conv_in, the residual blocks in order, after which block (index) each upsample (its conv) comes, conv_out, the latent statistics ([128] each), the two vocoders and the extension's STFT and mel bases ([514, 512] and [64, 257] F32 on the GPU).

gpu: M10.Gpu
input: M24.Conv2
blocks: Vec<M24.AudioRes>
ups: Vec<i64>
upConvs: Vec<M24.Conv2>
output: M24.Conv2
mean: Vec<f32>
std: Vec<f32>
vocoder: M20.BigVgan
bwe: M20.BigVgan
stft: M32.Tensor
mel: M32.Tensor

`AudioEncoder`

gizlang

type AudioEncoder = { gpu: Gpu; input: Conv2; blocks: Vec<AudioRes>; downs: Vec<i64>; downConvs: Vec<Conv2>; output: Conv2; mean: Vec<f32>; std: Vec<f32>; basis: Tensor; mel: Tensor; }

The audio encoder: conv_in, the residual blocks in order, after which block each downsample (its conv) comes, conv_out, the latent statistics, and the STFT and mel bases ([1026, 1024] and [64, 513] F32 on the GPU).

gpu: M10.Gpu
input: M24.Conv2
blocks: Vec<M24.AudioRes>
downs: Vec<i64>
downConvs: Vec<M24.Conv2>
output: M24.Conv2
mean: Vec<f32>
std: Vec<f32>
basis: M32.Tensor
mel: M32.Tensor

`Clip`

gizlang

type Clip = { c: i64; h: i64; w: i64; frames: Vec<bytes>; }

n frames of c planes of h x w bytes (RGB: c = 3; a mask: c = 1, 255 is 1).

c: i64
h: i64
w: i64
frames: Vec<bytes>

Functions

`load`

gizlang

function load(sys: System, path: string, gpu: Gpu, spare: i64): Ltx throws string

LTX-2.5 from ComfyUI's INT8 file path on gpu: its weights stay on the GPU (the blocks first, then the connectors and the AdaLN modules) while it keeps spare bytes free besides the slots, for a pass's rows (measured: 0.7 GB at 832 x 480 x 97).

Effects: fail, read, write, console, io.

`audioLatents`

gizlang

function audioLatents(frames: i64, rate: f64): i64

The audio latents of a video of frames frames at rate frames a second, as LTXVEmptyLatentAudio counts them: its seconds times 25 latents a second (16 kHz, hop 160, 4 mel frames a latent), rounded half to even.

Effects: fail.

`loadDurationHead`

gizlang

function loadDurationHead(sys: System, path: string, gpu: Gpu): DurationHead throws string

The duration head of the checkpoint at PATH on GPU.

Effects: fail, read, write, console, io.

`durationFrames`

gizlang

function durationFrames(seconds: f64, rate: f64, least: f64, most: f64): i64

seconds at rate frames a second as frames 1 + 8 k: clamped to least .. most seconds, then rounded down to the grid (up, when that falls short of the least).

Effects: fail.

`loadLora`

gizlang

function loadLora(sys: System, path: string, strength: f64, gpu: Gpu): Lora throws string

The LoRA in file path at strength on gpu.

Effects: fail, read, write, console, io.

`at`

gizlang

function at(sigma: f64, video: i64, audio: i64): Times

Times of a pass with every row of video video rows and audio audio rows at sigma.

Effects: fail.

`sharedText`

gizlang

function sharedText(sys: System, host: string, port: i64, prompt: string, authorization: string): Tensor throws string

The conditioning of prompt ([rows, 6144] F32 on the host, as Ltx.request takes it) from giz's shared LTX-2.5 text encoder at host:port (POST /api/ltx25/text-embeddings with the prompt and an empty negative prompt, and authorization when not empty; it answers with a ZIP of the two conditionings' safetensors). The request's bytes are those of its other clients, so its cache serves both.

Effects: fail, read, write, console, io.

`distilledSigmas`

gizlang

function distilledSigmas(): Vec<f64>

The noise levels of the distilled model's steps, from 1 to 0.

Effects: fail.

`noise`

gizlang

function noise(seed: u64, frames: i64, height: i64, width: i64, audio: i64): Vec<Tensor>

Seed seed's noise as ComfyUI's RandomNoise draws it for an audio and video latent (torch.randn of the video latent [1, 128, frames, height, width], then of the audio latent [1, 8, audio, 16], on one CPU generator), as rows: [video rows, audio rows].

Effects: fail, read, write, console, io.

`guidedNoise`

gizlang

function guidedNoise(seed: u64, frames: i64, height: i64, width: i64, audio: i64, guides: Vec<Guide>): Vec<Tensor>

noise() for a request with guides: drawn over the video's frames and the guides' at the full grid (as ComfyUI draws it over the latent with the guides appended), the video's rows kept and the guides' rows zero (each step sets them).

Effects: fail, read, write, console, io.

`videoLatent`

gizlang

function videoLatent(rows: Tensor, frames: i64, height: i64, width: i64): Tensor

A video latent's rows ([pixels, 128]) as the latent [128, frames, height, width] F32 on the host.

Effects: fail, read, write, console, io.

`audioLatent`

gizlang

function audioLatent(rows: Tensor, audio: i64): Tensor

An audio latent's rows ([audio, 128]) as the latent [8, audio, 16].

Effects: fail, read, write, console, io.

`loadVideoVae`

gizlang

function loadVideoVae(sys: System, path: string, gpu: Gpu): VideoVae throws string

The video decoder in ComfyUI's file path on gpu.

Effects: fail, read, write, console, io.

`loadDiffusionDecoder`

gizlang

function loadDiffusionDecoder(sys: System, path: string, gpu: Gpu): DiffusionDecoder throws string

The fidelity decoder in ComfyUI's file path on gpu.

Effects: fail, read, write, console, io.

`loadUpsampler`

gizlang

function loadUpsampler(sys: System, path: string, vae: string, gpu: Gpu): Upsampler throws string

The upsampler in ComfyUI's file path on gpu, with the latent statistics of the video VAE at vae.

Effects: fail, read, write, console, io.

`loadVideoEncoder`

gizlang

function loadVideoEncoder(sys: System, path: string, gpu: Gpu): VideoEncoder throws string

The encoder in ComfyUI's file path on gpu.

Effects: fail, read, write, console, io.

`guideFrames`

gizlang

function guideFrames(images: Vec<Image.Image>, width: i64, height: i64): Tensor

images as an encoder's frames of width x height pixels ([n, 3, height, width] F32 in [-1, 1]), each cut to the output's aspect about its center and resized by bilinear, as ComfyUI's LTXVAddGuide makes a guide's pixels.

Effects: fail.

`loadAudioVae`

gizlang

function loadAudioVae(sys: System, path: string, gpu: Gpu): AudioVae throws string

LTX-2.5's audio decoder and vocoders in ComfyUI's file path on gpu.

Effects: fail, read, write, console, io.

`loadAudioEncoder`

gizlang

function loadAudioEncoder(sys: System, path: string, gpu: Gpu): AudioEncoder throws string

The audio encoder in ComfyUI's audio VAE file path on gpu.

Effects: fail, read, write, console, io.

`clipOf`

gizlang

function clipOf(images: Vec<Image.Image>): Clip

The frames of RGB images (interleaved bytes) as a clip.

Effects: fail.

`clipOfBytes`

gizlang

function clipOfBytes(c: i64, h: i64, w: i64, frames: Vec<bytes>): Clip

A clip of frames of c planes of h x w bytes each.

`clipOfRgb`

gizlang

function clipOfRgb(h: i64, w: i64, frames: Vec<bytes>): Clip

A clip of interleaved RGB frames (ffmpeg's rgb24) of h x w.

Effects: fail.

`blendedFrame`

gizlang

function blendedFrame(a: Vec<f32>, b: bytes, mask: bytes, h: i64, w: i64, dilation: i64): Vec<f32>

ComfyUI's LTXVLaplacianPyramidBlend of frame a (generated, F32 [3, h, w] in [0, 1]) over frame b (bytes) by mask (one plane of bytes): the mask widened at a long side of 64 by dilation (bilinear down, max, bilinear up), the planes padded to powers of two, Laplacian pyramids of up to 7 levels mixed by the mask's Gaussian pyramid, cut back and clamped to [0, 1].

Effects: fail.

`blended`

gizlang

function blended(a: Tensor, b: Clip, mask: Clip, at: i64, dilation: i64): Tensor

blendedFrame of each frame of a ([k, 3, h, w] F32 in [0, 1] on a GPU) over frames at .. at + k of b by mask: the same on the GPU.

Effects: fail, read, write, console, io.

Methods

`Ltx.request`

gizlang

function Ltx.request(self: Ltx, texts: Vec<Tensor>, frames: i64, height: i64, width: i64, audio0: i64, rate: f64, guides: Vec<Guide>, reference: Tensor, lora: Vec<Lora>): Request throws string

A request for a video latent of frames x height x width (latent pixels) and audio audio latents at rate frames a second, conditioned on texts (Ltx.connect) and guided by guides (their frames' rows follow the video's). A soundtrack reference (its latent rows, AudioEncoder.encode; or none, [0, 128]) is the clip's audio, kept as it is, and the model also reads it as a voice before the clip (ComfyUI's LTXVSetAudioRefTokens).

Effects: fail, read, write, console, io.

`DurationHead.seconds`

gizlang

function DurationHead.seconds(self: DurationHead, texts: Vec<Tensor>): f64

The seconds the head predicts for connector rows texts (Ltx.connect).

Effects: fail, read, write, console, io.

`Ltx.connect`

gizlang

function Ltx.connect(self: Ltx, text: Tensor): Vec<Tensor> throws string

The text text ([rows, 6144]: video's 4096 values, then audio's 2048, as the shared encoder answers) through both connectors: [video rows, audio rows] as Ltx.request and the duration head take them.

Effects: fail, read, write, console, io.

`Ltx.free`

gizlang

function Ltx.free(self: Ltx, bytes: i64): void

Gives the GPU's memory back until bytes more fit: the AdaLN modules, the connectors, then the last blocks leave it, each staged first if it was not (the copy stays; a block's next use puts it back while there is room).

Effects: fail, read, write, io.

`Ltx.resident`

gizlang

function Ltx.resident(self: Ltx): i64

The blocks on the GPU now, of 48.

Effects: fail, read.

`Ltx.passes`

gizlang

function Ltx.passes(self: Ltx, list: Vec<Times>): Vec<Pass>

The modulation of passes at list, the AdaLN modules run once over all their time rows.

Effects: fail, read, write, console, io.

`Ltx.velocity`

gizlang

function Ltx.velocity(self: Ltx, r: Request, video: Tensor, audio: Tensor, p: Pass): Vec<Tensor>

The velocity of LTX-2.5 at video rows video ([rows, 128] F32 on the GPU: the latent's pixels in (frame, height, width) order) and audio rows audio ([rows, 128]: channel, then bin), modulated by pass p (Ltx.passes) of their times. [video velocity, audio velocity] as F32 rows.

Effects: fail, read, write, console, io.

`Ltx.sample`

gizlang

function Ltx.sample(self: Ltx, r: Request, start: Vec<Tensor>, sigmas: Vec<f64>, ancestral: bool, seed: u64, progress: (step: i64, total: i64) => bool, inPlace: Vec<InPlace> = []): Vec<Tensor>

Samples a request from noise rows (noise()) over sigmas with Euler steps, ancestral (eta 1: each step goes to sigma_next² / sigma, then renoises to sigma_next, as ComfyUI's euler_ancestral for a flow model) when ancestral, the renoising drawn from seed + 1 on the CPU generator, as ComfyUI does on a CPU (on a GPU it draws CUDA's, so its steps after the first differ from ours in their noise only). Every row is at the step's sigma but the guides' rows (the noise's last video rows): they stay the guides' clean latent at time 0, as ComfyUI's denoise mask keeps them. progress is told each step before it runs (its index and the steps): false stops there. With inPlace, its rows are at time sigma times its mask, the model sees them mixed with their noisy latent and its denoised rows are mixed with the latent by the mask (ComfyUI's KSamplerX0Inpaint). [video rows, audio rows] F32 on the GPU, the guides' rows left out.

Effects: fail, read, write, console, io.

`VideoVae.decode`

gizlang

function VideoVae.decode(self: VideoVae, latent: Tensor): Tensor

The frames of latent ([128, F, H, W] F32): [8 F - 7, 3, 32 H, 32 W] F32 on the GPU, in [-1, 1] (not clamped).

Effects: fail, read, write, console, io.

`VideoVae.tiles`

gizlang

function VideoVae.tiles(self: VideoVae, latent: Tensor): Tiles

The decoding of latent ([128, F, H, W] F32 on the host) in tiles: Tiles.next gives its frames in order.

Effects: fail, read, write, console, io.

`DiffusionDecoder.tiles`

gizlang

function DiffusionDecoder.tiles(self: DiffusionDecoder, latent: Tensor): Tiles

The fidelity decoder's tiles of latent, as VideoVae.tiles.

Effects: fail, read, write, console, io.

`Tiles.next`

gizlang

function Tiles.next(self: Tiles): Option<Tensor>

The next frames that are final, [k, 3, 32 H, 32 W] F32 on the GPU in [0, 1]: those of the next temporal tile before the one after it starts; none when all are given.

Effects: fail, read, write, console, io.

`DiffusionDecoder.decode`

gizlang

function DiffusionDecoder.decode(self: DiffusionDecoder, latent: Tensor): Tensor

The frames of latent ([128, F, H, W] normalized, on the host): [8 F - 7, 3, 32 H, 32 W] F32 on the GPU in [-1, 1].

Effects: fail, read, write, console, io.

`Upsampler.run`

gizlang

function Upsampler.run(self: Upsampler, latent: Tensor): Tensor

latent ([128, frames, h, w], normalized as the VAE's latents are) at twice its size.

Effects: fail, read, write, console, io.

`VideoEncoder.encode`

gizlang

function VideoEncoder.encode(self: VideoEncoder, frames: Tensor): Tensor

The latent ([128, k + 1, H / 32, W / 32] F32 on the host) of frames ([8 k + 1, 3, H, W] F32 in [-1, 1]).

Effects: fail, read, write, console, io.

`VideoEncoder.encodeFrom`

gizlang

function VideoEncoder.encodeFrom(self: VideoEncoder, n: i64, height: i64, width: i64, frames: (at: i64, m: i64) => Tensor): Tensor

encode() of n frames of H x W that frames gives a run at a time: frames at .. at + m ([m, 3, H, W] F32 in [-1, 1]), so the whole clip is never in memory.

Effects: fail, read, write, console, io.

`AudioVae.decode`

gizlang

function AudioVae.decode(self: AudioVae, rows: Tensor): Tensor

The 48 kHz stereo audio of audio latent rows ([T, 128]): [2, samples] F32 on the host, in [-1, 1].

Effects: fail, read, write, console, io.

`AudioEncoder.encode`

gizlang

function AudioEncoder.encode(self: AudioEncoder, left: Vec<f32>, right: Vec<f32>): Tensor

Stereo 16 kHz left and right as normalized latent rows [frames / 4, 128] on the GPU (channel, then bin).

Effects: fail, read, write, console, io.

`Clip.images`

gizlang

function Clip.images(self: Clip): Vec<Image.Image>

The frames as RGB images (interleaved bytes).

Effects: fail.

`Clip.red`

gizlang

function Clip.red(self: Clip): Clip

One plane of the clip's red values: a mask video as ComfyUI's ImageToMask (channel red) reads it.

Effects: fail.

`Clip.scaled`

gizlang

function Clip.scaled(self: Clip, oh: i64, ow: i64, mode: string, crop: bool): Clip

The clip resized to oh x ow by mode ("lanczos", "area", "bilinear", "nearest"), its center cut first to the new aspect when crop (ComfyUI's common_upscale).

Effects: fail.

`Clip.shorter`

gizlang

function Clip.shorter(self: Clip, side: i64, mode: string): Clip

ComfyUI's "scale shorter dimension" to side (the other side rounded half to even).

Effects: fail.

`Clip.toMultiple`

gizlang

function Clip.toMultiple(self: Clip, k: i64, mode: string): Clip

ComfyUI's "scale to multiple": the largest multiple of k each side, covered by a resize, then cut at the center.

Effects: fail.

`Clip.padded`

gizlang

function Clip.padded(self: Clip, width: i64, height: i64): Vec<Clip>

ComfyUI's ImagePadForOutpaintTargetSize (nearest-exact, no feathering): the frames shrunk to fit width x height if larger, centered on gray 0.5; and the mask, 1 in the padding. [frames, mask].

Effects: fail.

`Clip.green`

gizlang

function Clip.green(self: Clip, mask: Clip): Clip

The frames painted green where mask is set (LTXVInpaintPreprocess), the shorter of the two clips long.

Effects: fail.

`Clip.dilated`

gizlang

function Clip.dilated(self: Clip, radius: i64): Clip

A mask's frames dilated by radius pixels (max over the square around each) and cut at 0.5 (LTXVDilateVideoMask without time).

Effects: fail.

`Clip.pixels`

gizlang

function Clip.pixels(self: Clip, at: i64, m: i64): Tensor

Frames at .. at + m as [m, 3, h, w] F32 in [-1, 1] (VideoEncoder.encodeFrom).

Effects: fail.

`Clip.on`

gizlang

function Clip.on(self: Clip, gpu: Gpu, at: i64, m: i64): Tensor

The frames as F32 [n, 3, h, w] values in [0, 1] on gpu, frames at .. at + m.

Effects: fail, read, write, console, io.

`Clip.edges`

gizlang

function Clip.edges(self: Clip, low: f64, high: f64): Clip

Each frame's Canny edges as OpenCV's Canny (3 x 3 Sobel with the edge repeated, the largest gradient of the three colors, L1 magnitudes, thinned along the gradient, kept above high or above low when joined to one), white on black in all three planes: ComfyUI's CannyEdgePreprocessor at the frames' size.

Effects: fail.

Constants

`TOO_LARGE`

gizlang

const TOO_LARGE = "too large for this GPU"

What a request asks that does not fit this GPU (Ltx.request).

`VIDEO`

gizlang

const VIDEO = 4096

The width of a video row.

`AUDIO`

gizlang

const AUDIO = 2048

The width of an audio row.

`LATENT`

gizlang

const LATENT = 128

The channels of a video latent.

`AUDIO_CHANNELS`

gizlang

const AUDIO_CHANNELS = 8

The channels of an audio latent (a row is its channels times its frequency bins, 128 values).

`AUDIO_BINS`

gizlang

const AUDIO_BINS = 16

The frequency bins of an audio latent.

`FRAMES_A_LATENT`

gizlang

const FRAMES_A_LATENT = 8

The video frames of one latent frame of the video VAE.

`PIXELS_A_LATENT`

gizlang

const PIXELS_A_LATENT = 32

The pixels of one latent pixel of the video VAE, in height and in width.