# gizml:ltx Lightricks' LTX-2.5 (22B), the audio and video flow transformer of giz.ai's ltx2-5-distilled. Two streams of rows run side by side through 48 blocks: video (4096 wide, 32 heads of 128; a row a latent pixel of 8 frames and 32 x 32 pixels) and audio (2048 wide, 32 heads of 64; a row a 40 ms latent). In each block each stream attends to itself, then to its text, then the audio to the video and the video to the audio, then runs a GELU MLP; each step is modulated (AdaLN) by the stream's timestep, and the text's keys and values by the largest one. Attention reads q and k over their RMS across all heads, then rotated (RoPE by split halves) by the rows' positions (video: time, height, width; audio: time), and its output is gated a head by the input's linear. A text's conditioning (Gemma4's hidden states projected to [rows, 4096 + 2048], the shared encoder's answer) first runs through each stream's connector: 8 blocks of self attention over the rows padded to 1024 with learned registers. The linears are INT8 rows rotated by ConvRot's Hadamard of 256, a scale a row (ComfyUI's ltx-2.5-22b-distilled-transformer-comfy-int8-convrot file, read as it is: gizml:diffusion's comfyInt8Rows), run on the INT8 matrix cores against x quantized as N.int8Rows quantizes it. Every weight block is in page-locked host memory (gizml:diffusion's Staged) and, while the card has room, on the GPU: the blocks first (each step reads them), then the connectors and the AdaLN modules (a request reads them once: every step's modulation is made at its start). The blocks that stay are spread over the 48, and the others copy to one of two slots, each as soon as its slot is free, so the copies run all through a pass. Ltx.free gives memory back for the VAEs; a block that copies for its use stays on the GPU again while there is room, so the next request gets it back at no cost. On a 16 GB card the copies bound a step, so every byte left on the card counts. The residual streams are F32; every product's output rounds to BF16, as ComfyUI stores it. ```gizlang import * as Ltx from "gizml:ltx"; ``` ## Types ### `Dense` ```gizlang type Dense = { w: Tensor; b: Tensor; } ``` A linear with BF16 weights ([out, in] on the GPU) and its bias ([1, out] F32; [0] for none). - `w`: `M32.Tensor` - `b`: `M32.Tensor` ### `Weights` ```gizlang type Weights = { staged: Ref>>; bytes: Vec; resident: Ref>>; wanted: Vec; } ``` Blocks in order: each on the GPU while it stays there (resident), in page-locked host memory (staged) once it is off the GPU (a block resident at load is staged when Ltx.free takes it off, so the host holds only what the card does not), its staged bytes, and whether it is one to keep there (wanted: the ones that stay are spread over the blocks, so the copies of the others run while the GPU computes). Each block not resident is staged. - `staged`: `Ref>>` - `bytes`: `Vec` - `resident`: `Ref>>` - `wanted`: `Vec` ### `Ltx` ```gizlang type Ltx = { gpu: Gpu; adaln: Weights; videoIn: Dense; audioIn: Dense; videoOut: Dense; audioOut: Dense; outTables: Vec; marker: Tensor; connectors: Vec; registers: Vec; blocks: Weights; slots: Vec; spare: Ref; least: i64; tables: Vec; ones: Vec; } ``` LTX-2.5 where it runs: its AdaLN modules (8), the input and output projections (video, audio), the output tables ([1, 2, width]), the first frame's marker, the two connectors and their registers, the blocks, the two slots of the blocks that copy, the bytes a pass needs besides the weights (spare: the request's; least: the load's), and the blocks' tables (T_VIDEO... order, [blocks, rows, width]); every table F32 of BF16 values on the GPU. - `gpu`: `M10.Gpu` - `adaln`: `M24.Weights` - `videoIn`: `M24.Dense` - `audioIn`: `M24.Dense` - `videoOut`: `M24.Dense` - `audioOut`: `M24.Dense` - `outTables`: `Vec` - `marker`: `M32.Tensor` - `connectors`: `Vec` - `registers`: `Vec` - `blocks`: `M24.Weights` - `slots`: `Vec` - `spare`: `Ref` - `least`: `i64` - `tables`: `Vec` - `ones`: `Vec` ### `Request` ```gizlang type Request = { frames: i64; height: i64; width: i64; audio: i64; texts: Vec; videoRope: Vec; audioRope: Vec; videoCross: Vec; audioCross: Vec; guides: Tensor; reference: Tensor; lora: Vec; } ``` A request's fixed parts: its latent grid (frames, height, width of the video latent) and audio latents, the two streams' texts after their connectors ([TEXT_ROWS+, width] BF16), and the rotary tables of each attention (video self, audio self, the cross attentions' video and audio rows). - `frames`: `i64` - `height`: `i64` - `width`: `i64` - `audio`: `i64` - `texts`: `Vec` - `videoRope`: `Vec` - `audioRope`: `Vec` - `videoCross`: `Vec` - `audioCross`: `Vec` - `guides`: `M32.Tensor` - `reference`: `M32.Tensor` - `lora`: `Vec` ### `Guide` ```gizlang type Guide = { latent: Tensor; frame: i64; scale: i64; } ``` A clean latent that guides a request (LTXVAddGuide at strength 1): its frames ([128, f, H / scale, W / scale] F32, as VideoEncoder.encode gives them) from pixel frame `frame` on. A `scale` of 2 is an IC-LoRA's small grid (LTXAddVideoICLoRAGuide's latent_downscale_factor): each row stands for scale x scale latent pixels. - `latent`: `M32.Tensor` - `frame`: `i64` - `scale`: `i64` ### `DurationHead` ```gizlang type DurationHead = { video: Dense; audio: Dense; videoEmb: Vec; audioEmb: Vec; query: Vec; kv: Tensor; kvBias: Tensor; inW: Vec; inB: Vec; outW: Vec; outB: Vec; hiddenW: Vec; hiddenB: Vec; lastW: Vec; lastB: Vec; } ``` giz's autoDuration: LTX's duration head (ComfyUI's DurationHead) over a request's connector rows. Each stream's rows go to 256 values (plus their stream's embedding); one learned query pools them in 4 heads of 64; an MLP (GELU, tanh form) gives the log of the seconds. - `video`: `M24.Dense` - `audio`: `M24.Dense` - `videoEmb`: `Vec` - `audioEmb`: `Vec` - `query`: `Vec` - `kv`: `M32.Tensor` - `kvBias`: `M32.Tensor` - `inW`: `Vec` - `inB`: `Vec` - `outW`: `Vec` - `outB`: `Vec` - `hiddenW`: `Vec` - `hiddenB`: `Vec` - `lastW`: `Vec` - `lastB`: `Vec` ### `Lora` ```gizlang type Lora = { a: Vec>; b: Vec>; scale: i64; } ``` An IC-LoRA (giz's ingredients, in/outpainting and control workflows): low-rank updates B A, times `strength`, of the video stream's linears in each block (self attention's q, k, v and out, the text's, the MLP's two). For each block and linear (V_QKV .. V_DOWN): A [parts r, in] (its parts' A stacked) and B [out, parts r] (their B block-diagonal, the strength in it), BF16 on the GPU. `scale` is how much smaller its guide latents are (reference_downscale_factor). - `a`: `Vec>` - `b`: `Vec>` - `scale`: `i64` ### `Pass` ```gizlang type Pass = { video: Vec>; audio: Vec>; videoMap: Tensor; audioMap: Tensor; videoRows: Vec; audioRows: Vec; videoOut: Vec; audioOut: Vec; } ``` The modulation of one pass where the GPU reads it: for each block, rows of its parameters (Mods' order) at each time row of its stream, [times, width] F32. - `video`: `Vec>` - `audio`: `Vec>` - `videoMap`: `M32.Tensor` - `audioMap`: `M32.Tensor` - `videoRows`: `Vec` - `audioRows`: `Vec` - `videoOut`: `Vec` - `audioOut`: `Vec` ### `Times` ```gizlang type Times = { video: Vec; videoMap: Vec; audio: Vec; audioMap: Vec; } ``` A pass's times: each stream's time rows (sigma in [0, 1]) and the time row of each of its rows. - `video`: `Vec` - `videoMap`: `Vec` - `audio`: `Vec` - `audioMap`: `Vec` ### `InPlace` ```gizlang type InPlace = { latent: Tensor; noise: Tensor; mask: f64; } ``` The first latent frame held in place (ComfyUI's LTXVImgToVideoInplace): its rows `latent` ([k, 128] F32, the video's first k rows), the start's noise of those rows (unscaled), and the denoise mask 1 - strength. - `latent`: `M32.Tensor` - `noise`: `M32.Tensor` - `mask`: `f64` ### `Conv3` ```gizlang type Conv3 = { w: Tensor; b: Tensor; } ``` A 3 x 3 x 3 conv: its weight [out, 27 in] BF16 in that order, its bias [out, 1] F32, on the GPU. - `w`: `M32.Tensor` - `b`: `M32.Tensor` ### `VaeBlock` ```gizlang type VaeBlock = { convs: Vec; cell: Vec; } ``` A decoder block: residual pairs (convs in pairs) or an upsample (one conv, its cell's time, height and width). - `convs`: `Vec` - `cell`: `Vec` ### `VideoVae` ```gizlang type VideoVae = { gpu: Gpu; input: Conv3; blocks: Vec; output: Conv3; mean: Vec; std: Vec; } ``` The decoder where it runs: conv_in, the blocks, conv_out, and each latent channel's mean and deviation. - `gpu`: `M10.Gpu` - `input`: `M24.Conv3` - `blocks`: `Vec` - `output`: `M24.Conv3` - `mean`: `Vec` - `std`: `Vec` ### `Tiles` ```gizlang type Tiles = { gpu: Gpu; vae: Vec; fidelity: Vec; z: Tensor; ts: Vec; ys: Vec; xs: Vec; wt: Vec>; wy: Vec>; wx: Vec>; next: Ref; window: Ref; base: Ref; } ``` A decoding in tiles (VideoVae.tiles): the latent, each side's tile starts and weights, the next temporal tile, and the output of the last one from its first frame (`base`). - `gpu`: `M10.Gpu` - `vae`: `Vec` - `fidelity`: `Vec` - `z`: `M32.Tensor` - `ts`: `Vec` - `ys`: `Vec` - `xs`: `Vec` - `wt`: `Vec>` - `wy`: `Vec>` - `wx`: `Vec>` - `next`: `Ref` - `window`: `Ref` - `base`: `Ref` ### `NaAttention` ```gizlang type NaAttention = { kernel: Vec; qkv: Tensor; bias: Tensor; proj: Dense; qw: Tensor; kw: Tensor; } ``` A neighborhood attention: its window, qkv (q and k rows within each head in split-half order: the file's pairs 2 i, 2 i + 1 become i, i + 32, so gizml:netops' split-half rotary turns them), its bias, the output linear, and the q and k RMSNorm weights (`[head]` F32, the same order; q's times head^-1/2). - `kernel`: `Vec` - `qkv`: `M32.Tensor` - `bias`: `M32.Tensor` - `proj`: `M24.Dense` - `qw`: `M32.Tensor` - `kw`: `M32.Tensor` ### `NaBlock` ```gizlang type NaBlock = { norm1: Tensor; attention: NaAttention; norm2: Tensor; up: Tensor; gate: Tensor; down: Tensor; context: Vec; mods: Vec; } ``` A block: its norms (`[c]` F32), attention, SwiGLU (up, gate, down), and in the diffusion stage its context linear and the step's shift and scale of each norm ([1, c] F32: the AdaLN plus its table). - `norm1`: `M32.Tensor` - `attention`: `M24.NaAttention` - `norm2`: `M32.Tensor` - `up`: `M32.Tensor` - `gate`: `M32.Tensor` - `down`: `M32.Tensor` - `context`: `Vec` - `mods`: `Vec` ### `NaUpsample` ```gizlang type NaUpsample = { proj: Dense; stride: Vec; } ``` A linear pixel shuffle: its linear and stride (time, height, width). - `proj`: `M24.Dense` - `stride`: `Vec` ### `DiffusionDecoder` ```gizlang type DiffusionDecoder = { gpu: Gpu; input: Dense; stages: Vec>; ups: Vec; noisy: Dense; blocks: Vec; norm: Tensor; output: Dense; mean: Vec; std: Vec; head: i64; patch: i64; noise: Ref; } ``` The diffusion decoder of the video VAE: its stages, upsamplers, blocks and normalization. - `gpu`: `M10.Gpu` - `input`: `M24.Dense` - `stages`: `Vec>` - `ups`: `Vec` - `noisy`: `M24.Dense` - `blocks`: `Vec` - `norm`: `M32.Tensor` - `output`: `M24.Dense` - `mean`: `Vec` - `std`: `Vec` - `head`: `i64` - `patch`: `i64` - `noise`: `Ref` ### `Upsampler` ```gizlang type Upsampler = { gpu: Gpu; input: Conv3; inNorm: Vec; convs: Vec; norms: Vec>; up: Tensor; upBias: Tensor; output: Conv3; mean: Vec; std: Vec; } ``` giz's quality mode doubles the first stage's latent in space (ComfyUI's LatentUpsampler under LTXVLatentUpsampler): on the latent without its normalization, a 3D conv to 1024 channels, GroupNorm (32 groups) and SiLU, 4 residual blocks, a 2D conv to 4 times the channels a frame and a pixel shuffle, 4 residual blocks, a 3D conv back to 128; then normalized again. Its 3D convs pad with zeros in time as in space. - `gpu`: `M10.Gpu` - `input`: `M24.Conv3` - `inNorm`: `Vec` - `convs`: `Vec` - `norms`: `Vec>` - `up`: `M32.Tensor` - `upBias`: `M32.Tensor` - `output`: `M24.Conv3` - `mean`: `Vec` - `std`: `Vec` ### `VideoEncoder` ```gizlang type VideoEncoder = { gpu: Gpu; input: Conv3; blocks: Vec; output: Conv3; mean: Vec; std: Vec; } ``` The encoder where it runs: conv_in, the blocks (a downsample's cell its stride), conv_out, the latent's statistics. - `gpu`: `M10.Gpu` - `input`: `M24.Conv3` - `blocks`: `Vec` - `output`: `M24.Conv3` - `mean`: `Vec` - `std`: `Vec` ### `Conv2` ```gizlang type Conv2 = { w: Tensor; b: Tensor; size: i64; } ``` A 2D conv: its weight [out, in k k] F32 and bias [out, 1] F32, on the GPU. - `w`: `M32.Tensor` - `b`: `M32.Tensor` - `size`: `i64` ### `AudioRes` ```gizlang type AudioRes = { convs: Vec; } ``` A residual block: conv1, conv2 and the 1 x 1 shortcut where the channels change. - `convs`: `Vec` ### `AudioVae` ```gizlang type AudioVae = { gpu: Gpu; input: Conv2; blocks: Vec; ups: Vec; upConvs: Vec; output: Conv2; mean: Vec; std: Vec; vocoder: D.BigVgan; bwe: D.BigVgan; stft: Tensor; mel: Tensor; } ``` The audio decoder: conv_in, the residual blocks in order, after which block (index) each upsample (its conv) comes, conv_out, the latent statistics ([128] each), the two vocoders and the extension's STFT and mel bases ([514, 512] and [64, 257] F32 on the GPU). - `gpu`: `M10.Gpu` - `input`: `M24.Conv2` - `blocks`: `Vec` - `ups`: `Vec` - `upConvs`: `Vec` - `output`: `M24.Conv2` - `mean`: `Vec` - `std`: `Vec` - `vocoder`: `M20.BigVgan` - `bwe`: `M20.BigVgan` - `stft`: `M32.Tensor` - `mel`: `M32.Tensor` ### `AudioEncoder` ```gizlang type AudioEncoder = { gpu: Gpu; input: Conv2; blocks: Vec; downs: Vec; downConvs: Vec; output: Conv2; mean: Vec; std: Vec; basis: Tensor; mel: Tensor; } ``` The audio encoder: conv_in, the residual blocks in order, after which block each downsample (its conv) comes, conv_out, the latent statistics, and the STFT and mel bases ([1026, 1024] and [64, 513] F32 on the GPU). - `gpu`: `M10.Gpu` - `input`: `M24.Conv2` - `blocks`: `Vec` - `downs`: `Vec` - `downConvs`: `Vec` - `output`: `M24.Conv2` - `mean`: `Vec` - `std`: `Vec` - `basis`: `M32.Tensor` - `mel`: `M32.Tensor` ### `Clip` ```gizlang type Clip = { c: i64; h: i64; w: i64; frames: Vec; } ``` n frames of c planes of h x w bytes (RGB: c = 3; a mask: c = 1, 255 is 1). - `c`: `i64` - `h`: `i64` - `w`: `i64` - `frames`: `Vec` ## Functions ### `load` ```gizlang function load(sys: System, path: string, gpu: Gpu, spare: i64): Ltx throws string ``` LTX-2.5 from ComfyUI's INT8 file `path` on `gpu`: its weights stay on the GPU (the blocks first, then the connectors and the AdaLN modules) while it keeps `spare` bytes free besides the slots, for a pass's rows (measured: 0.7 GB at 832 x 480 x 97). Effects: fail, read, write, console, io. ### `audioLatents` ```gizlang function audioLatents(frames: i64, rate: f64): i64 ``` The audio latents of a video of `frames` frames at `rate` frames a second, as LTXVEmptyLatentAudio counts them: its seconds times 25 latents a second (16 kHz, hop 160, 4 mel frames a latent), rounded half to even. Effects: fail. ### `loadDurationHead` ```gizlang function loadDurationHead(sys: System, path: string, gpu: Gpu): DurationHead throws string ``` The duration head of the checkpoint at PATH on GPU. Effects: fail, read, write, console, io. ### `durationFrames` ```gizlang function durationFrames(seconds: f64, rate: f64, least: f64, most: f64): i64 ``` `seconds` at `rate` frames a second as frames 1 + 8 k: clamped to `least` .. `most` seconds, then rounded down to the grid (up, when that falls short of the least). Effects: fail. ### `loadLora` ```gizlang function loadLora(sys: System, path: string, strength: f64, gpu: Gpu): Lora throws string ``` The LoRA in file `path` at `strength` on `gpu`. Effects: fail, read, write, console, io. ### `at` ```gizlang function at(sigma: f64, video: i64, audio: i64): Times ``` Times of a pass with every row of `video` video rows and `audio` audio rows at `sigma`. Effects: fail. ### `sharedText` ```gizlang function sharedText(sys: System, host: string, port: i64, prompt: string, authorization: string): Tensor throws string ``` The conditioning of `prompt` ([rows, 6144] F32 on the host, as Ltx.request takes it) from giz's shared LTX-2.5 text encoder at host:port (POST /api/ltx25/text-embeddings with the prompt and an empty negative prompt, and `authorization` when not empty; it answers with a ZIP of the two conditionings' safetensors). The request's bytes are those of its other clients, so its cache serves both. Effects: fail, read, write, console, io. ### `distilledSigmas` ```gizlang function distilledSigmas(): Vec ``` The noise levels of the distilled model's steps, from 1 to 0. Effects: fail. ### `noise` ```gizlang function noise(seed: u64, frames: i64, height: i64, width: i64, audio: i64): Vec ``` Seed `seed`'s noise as ComfyUI's RandomNoise draws it for an audio and video latent (torch.randn of the video latent [1, 128, frames, height, width], then of the audio latent [1, 8, audio, 16], on one CPU generator), as rows: [video rows, audio rows]. Effects: fail, read, write, console, io. ### `guidedNoise` ```gizlang function guidedNoise(seed: u64, frames: i64, height: i64, width: i64, audio: i64, guides: Vec): Vec ``` noise() for a request with `guides`: drawn over the video's frames and the guides' at the full grid (as ComfyUI draws it over the latent with the guides appended), the video's rows kept and the guides' rows zero (each step sets them). Effects: fail, read, write, console, io. ### `videoLatent` ```gizlang function videoLatent(rows: Tensor, frames: i64, height: i64, width: i64): Tensor ``` A video latent's rows ([pixels, 128]) as the latent [128, frames, height, width] F32 on the host. Effects: fail, read, write, console, io. ### `audioLatent` ```gizlang function audioLatent(rows: Tensor, audio: i64): Tensor ``` An audio latent's rows ([audio, 128]) as the latent [8, audio, 16]. Effects: fail, read, write, console, io. ### `loadVideoVae` ```gizlang function loadVideoVae(sys: System, path: string, gpu: Gpu): VideoVae throws string ``` The video decoder in ComfyUI's file `path` on `gpu`. Effects: fail, read, write, console, io. ### `loadDiffusionDecoder` ```gizlang function loadDiffusionDecoder(sys: System, path: string, gpu: Gpu): DiffusionDecoder throws string ``` The fidelity decoder in ComfyUI's file `path` on `gpu`. Effects: fail, read, write, console, io. ### `loadUpsampler` ```gizlang function loadUpsampler(sys: System, path: string, vae: string, gpu: Gpu): Upsampler throws string ``` The upsampler in ComfyUI's file `path` on `gpu`, with the latent statistics of the video VAE at `vae`. Effects: fail, read, write, console, io. ### `loadVideoEncoder` ```gizlang function loadVideoEncoder(sys: System, path: string, gpu: Gpu): VideoEncoder throws string ``` The encoder in ComfyUI's file `path` on `gpu`. Effects: fail, read, write, console, io. ### `guideFrames` ```gizlang function guideFrames(images: Vec, width: i64, height: i64): Tensor ``` `images` as an encoder's frames of width x height pixels ([n, 3, height, width] F32 in [-1, 1]), each cut to the output's aspect about its center and resized by bilinear, as ComfyUI's LTXVAddGuide makes a guide's pixels. Effects: fail. ### `loadAudioVae` ```gizlang function loadAudioVae(sys: System, path: string, gpu: Gpu): AudioVae throws string ``` LTX-2.5's audio decoder and vocoders in ComfyUI's file `path` on `gpu`. Effects: fail, read, write, console, io. ### `loadAudioEncoder` ```gizlang function loadAudioEncoder(sys: System, path: string, gpu: Gpu): AudioEncoder throws string ``` The audio encoder in ComfyUI's audio VAE file `path` on `gpu`. Effects: fail, read, write, console, io. ### `clipOf` ```gizlang function clipOf(images: Vec): Clip ``` The frames of RGB images (interleaved bytes) as a clip. Effects: fail. ### `clipOfBytes` ```gizlang function clipOfBytes(c: i64, h: i64, w: i64, frames: Vec): Clip ``` A clip of frames of c planes of h x w bytes each. ### `clipOfRgb` ```gizlang function clipOfRgb(h: i64, w: i64, frames: Vec): Clip ``` A clip of interleaved RGB frames (ffmpeg's rgb24) of h x w. Effects: fail. ### `blendedFrame` ```gizlang function blendedFrame(a: Vec, b: bytes, mask: bytes, h: i64, w: i64, dilation: i64): Vec ``` ComfyUI's LTXVLaplacianPyramidBlend of frame a (generated, F32 [3, h, w] in [0, 1]) over frame b (bytes) by `mask` (one plane of bytes): the mask widened at a long side of 64 by `dilation` (bilinear down, max, bilinear up), the planes padded to powers of two, Laplacian pyramids of up to 7 levels mixed by the mask's Gaussian pyramid, cut back and clamped to [0, 1]. Effects: fail. ### `blended` ```gizlang function blended(a: Tensor, b: Clip, mask: Clip, at: i64, dilation: i64): Tensor ``` blendedFrame of each frame of `a` ([k, 3, h, w] F32 in [0, 1] on a GPU) over frames at .. at + k of `b` by `mask`: the same on the GPU. Effects: fail, read, write, console, io. ## Methods ### `Ltx.request` ```gizlang function Ltx.request(self: Ltx, texts: Vec, frames: i64, height: i64, width: i64, audio0: i64, rate: f64, guides: Vec, reference: Tensor, lora: Vec): Request throws string ``` A request for a video latent of `frames` x `height` x `width` (latent pixels) and `audio` audio latents at `rate` frames a second, conditioned on `texts` (Ltx.connect) and guided by `guides` (their frames' rows follow the video's). A soundtrack `reference` (its latent rows, AudioEncoder.encode; or none, [0, 128]) is the clip's audio, kept as it is, and the model also reads it as a voice before the clip (ComfyUI's LTXVSetAudioRefTokens). Effects: fail, read, write, console, io. ### `DurationHead.seconds` ```gizlang function DurationHead.seconds(self: DurationHead, texts: Vec): f64 ``` The seconds the head predicts for connector rows `texts` (Ltx.connect). Effects: fail, read, write, console, io. ### `Ltx.connect` ```gizlang function Ltx.connect(self: Ltx, text: Tensor): Vec throws string ``` The text `text` ([rows, 6144]: video's 4096 values, then audio's 2048, as the shared encoder answers) through both connectors: [video rows, audio rows] as Ltx.request and the duration head take them. Effects: fail, read, write, console, io. ### `Ltx.free` ```gizlang function Ltx.free(self: Ltx, bytes: i64): void ``` Gives the GPU's memory back until `bytes` more fit: the AdaLN modules, the connectors, then the last blocks leave it, each staged first if it was not (the copy stays; a block's next use puts it back while there is room). Effects: fail, read, write, io. ### `Ltx.resident` ```gizlang function Ltx.resident(self: Ltx): i64 ``` The blocks on the GPU now, of 48. Effects: fail, read. ### `Ltx.passes` ```gizlang function Ltx.passes(self: Ltx, list: Vec): Vec ``` The modulation of passes at `list`, the AdaLN modules run once over all their time rows. Effects: fail, read, write, console, io. ### `Ltx.velocity` ```gizlang function Ltx.velocity(self: Ltx, r: Request, video: Tensor, audio: Tensor, p: Pass): Vec ``` The velocity of LTX-2.5 at video rows `video` ([rows, 128] F32 on the GPU: the latent's pixels in (frame, height, width) order) and audio rows `audio` ([rows, 128]: channel, then bin), modulated by pass p (Ltx.passes) of their times. [video velocity, audio velocity] as F32 rows. Effects: fail, read, write, console, io. ### `Ltx.sample` ```gizlang function Ltx.sample(self: Ltx, r: Request, start: Vec, sigmas: Vec, ancestral: bool, seed: u64, progress: (step: i64, total: i64) => bool, inPlace: Vec = []): Vec ``` Samples a request from noise rows (noise()) over `sigmas` with Euler steps, ancestral (eta 1: each step goes to sigma_next² / sigma, then renoises to sigma_next, as ComfyUI's euler_ancestral for a flow model) when `ancestral`, the renoising drawn from `seed` + 1 on the CPU generator, as ComfyUI does on a CPU (on a GPU it draws CUDA's, so its steps after the first differ from ours in their noise only). Every row is at the step's sigma but the guides' rows (the noise's last video rows): they stay the guides' clean latent at time 0, as ComfyUI's denoise mask keeps them. `progress` is told each step before it runs (its index and the steps): false stops there. With `inPlace`, its rows are at time sigma times its mask, the model sees them mixed with their noisy latent and its denoised rows are mixed with the latent by the mask (ComfyUI's KSamplerX0Inpaint). [video rows, audio rows] F32 on the GPU, the guides' rows left out. Effects: fail, read, write, console, io. ### `VideoVae.decode` ```gizlang function VideoVae.decode(self: VideoVae, latent: Tensor): Tensor ``` The frames of `latent` ([128, F, H, W] F32): [8 F - 7, 3, 32 H, 32 W] F32 on the GPU, in [-1, 1] (not clamped). Effects: fail, read, write, console, io. ### `VideoVae.tiles` ```gizlang function VideoVae.tiles(self: VideoVae, latent: Tensor): Tiles ``` The decoding of `latent` ([128, F, H, W] F32 on the host) in tiles: Tiles.next gives its frames in order. Effects: fail, read, write, console, io. ### `DiffusionDecoder.tiles` ```gizlang function DiffusionDecoder.tiles(self: DiffusionDecoder, latent: Tensor): Tiles ``` The fidelity decoder's tiles of `latent`, as VideoVae.tiles. Effects: fail, read, write, console, io. ### `Tiles.next` ```gizlang function Tiles.next(self: Tiles): Option ``` The next frames that are final, [k, 3, 32 H, 32 W] F32 on the GPU in [0, 1]: those of the next temporal tile before the one after it starts; none when all are given. Effects: fail, read, write, console, io. ### `DiffusionDecoder.decode` ```gizlang function DiffusionDecoder.decode(self: DiffusionDecoder, latent: Tensor): Tensor ``` The frames of `latent` ([128, F, H, W] normalized, on the host): [8 F - 7, 3, 32 H, 32 W] F32 on the GPU in [-1, 1]. Effects: fail, read, write, console, io. ### `Upsampler.run` ```gizlang function Upsampler.run(self: Upsampler, latent: Tensor): Tensor ``` `latent` ([128, frames, h, w], normalized as the VAE's latents are) at twice its size. Effects: fail, read, write, console, io. ### `VideoEncoder.encode` ```gizlang function VideoEncoder.encode(self: VideoEncoder, frames: Tensor): Tensor ``` The latent ([128, k + 1, H / 32, W / 32] F32 on the host) of `frames` ([8 k + 1, 3, H, W] F32 in [-1, 1]). Effects: fail, read, write, console, io. ### `VideoEncoder.encodeFrom` ```gizlang function VideoEncoder.encodeFrom(self: VideoEncoder, n: i64, height: i64, width: i64, frames: (at: i64, m: i64) => Tensor): Tensor ``` encode() of n frames of H x W that `frames` gives a run at a time: frames at .. at + m ([m, 3, H, W] F32 in [-1, 1]), so the whole clip is never in memory. Effects: fail, read, write, console, io. ### `AudioVae.decode` ```gizlang function AudioVae.decode(self: AudioVae, rows: Tensor): Tensor ``` The 48 kHz stereo audio of audio latent rows ([T, 128]): [2, samples] F32 on the host, in [-1, 1]. Effects: fail, read, write, console, io. ### `AudioEncoder.encode` ```gizlang function AudioEncoder.encode(self: AudioEncoder, left: Vec, right: Vec): Tensor ``` Stereo 16 kHz `left` and `right` as normalized latent rows [frames / 4, 128] on the GPU (channel, then bin). Effects: fail, read, write, console, io. ### `Clip.images` ```gizlang function Clip.images(self: Clip): Vec ``` The frames as RGB images (interleaved bytes). Effects: fail. ### `Clip.red` ```gizlang function Clip.red(self: Clip): Clip ``` One plane of the clip's red values: a mask video as ComfyUI's ImageToMask (channel red) reads it. Effects: fail. ### `Clip.scaled` ```gizlang function Clip.scaled(self: Clip, oh: i64, ow: i64, mode: string, crop: bool): Clip ``` The clip resized to oh x ow by mode ("lanczos", "area", "bilinear", "nearest"), its center cut first to the new aspect when `crop` (ComfyUI's common_upscale). Effects: fail. ### `Clip.shorter` ```gizlang function Clip.shorter(self: Clip, side: i64, mode: string): Clip ``` ComfyUI's "scale shorter dimension" to `side` (the other side rounded half to even). Effects: fail. ### `Clip.toMultiple` ```gizlang function Clip.toMultiple(self: Clip, k: i64, mode: string): Clip ``` ComfyUI's "scale to multiple": the largest multiple of `k` each side, covered by a resize, then cut at the center. Effects: fail. ### `Clip.padded` ```gizlang function Clip.padded(self: Clip, width: i64, height: i64): Vec ``` ComfyUI's ImagePadForOutpaintTargetSize (nearest-exact, no feathering): the frames shrunk to fit width x height if larger, centered on gray 0.5; and the mask, 1 in the padding. [frames, mask]. Effects: fail. ### `Clip.green` ```gizlang function Clip.green(self: Clip, mask: Clip): Clip ``` The frames painted green where `mask` is set (LTXVInpaintPreprocess), the shorter of the two clips long. Effects: fail. ### `Clip.dilated` ```gizlang function Clip.dilated(self: Clip, radius: i64): Clip ``` A mask's frames dilated by `radius` pixels (max over the square around each) and cut at 0.5 (LTXVDilateVideoMask without time). Effects: fail. ### `Clip.pixels` ```gizlang function Clip.pixels(self: Clip, at: i64, m: i64): Tensor ``` Frames at .. at + m as [m, 3, h, w] F32 in [-1, 1] (VideoEncoder.encodeFrom). Effects: fail. ### `Clip.on` ```gizlang function Clip.on(self: Clip, gpu: Gpu, at: i64, m: i64): Tensor ``` The frames as F32 [n, 3, h, w] values in [0, 1] on `gpu`, frames at .. at + m. Effects: fail, read, write, console, io. ### `Clip.edges` ```gizlang function Clip.edges(self: Clip, low: f64, high: f64): Clip ``` Each frame's Canny edges as OpenCV's Canny (3 x 3 Sobel with the edge repeated, the largest gradient of the three colors, L1 magnitudes, thinned along the gradient, kept above `high` or above `low` when joined to one), white on black in all three planes: ComfyUI's CannyEdgePreprocessor at the frames' size. Effects: fail. ## Constants ### `TOO_LARGE` ```gizlang const TOO_LARGE = "too large for this GPU" ``` What a request asks that does not fit this GPU (Ltx.request). ### `VIDEO` ```gizlang const VIDEO = 4096 ``` The width of a video row. ### `AUDIO` ```gizlang const AUDIO = 2048 ``` The width of an audio row. ### `LATENT` ```gizlang const LATENT = 128 ``` The channels of a video latent. ### `AUDIO_CHANNELS` ```gizlang const AUDIO_CHANNELS = 8 ``` The channels of an audio latent (a row is its channels times its frequency bins, 128 values). ### `AUDIO_BINS` ```gizlang const AUDIO_BINS = 16 ``` The frequency bins of an audio latent. ### `FRAMES_A_LATENT` ```gizlang const FRAMES_A_LATENT = 8 ``` The video frames of one latent frame of the video VAE. ### `PIXELS_A_LATENT` ```gizlang const PIXELS_A_LATENT = 32 ``` The pixels of one latent pixel of the video VAE, in height and in width.