# gizml:video

video on top of gizhw:codec and gizlang:mp4. H.264 tracks of frame tensors (NVENC, or x264 through ffmpeg on other GPUs), NVIDIA's RTX Video effects and frame interpolation, DWPose's pose frames, an AAC-LC encoder on the host, WAV files and resampling, and any audio file and the frames of any video file through ffmpeg.

```gizlang
import * as Video from "gizml:video";
```

## Types

### `Enhance`

```gizlang
type Enhance = { effect: string; size: string; quality: string; scale: i64; }
```

What the enhancement does, as giz's GizRTXVideoEnhance node takes it. `effect`: SUPER_RESOLUTION (for compressed
video), HIGH_BITRATE (for clean video), DENOISE, DEBLUR or BICUBIC (no AI); `size`: HD, FHD, 2k or 4k (the long side
1280, 1920, 2560 or 3840, never below the source's); `quality`: LOW, MEDIUM, HIGH or ULTRA; `scale`: 1 to 4 times
the source's sides in place of `size` (0: by size). DENOISE and DEBLUR keep the source's size.

- `effect`: `string`
- `size`: `string`
- `quality`: `string`
- `scale`: `i64`

### `Effect`

```gizlang
type Effect = { nat: Native; fx: Library; image: Library; handle: u64; scratch: u64; src: u64; dst: u64; }
```

The effect's libraries, an effect made and loaded, and its two images (NvCVImage, 64 bytes each) on GPU buffers.

- `nat`: `gizlang_native.Native`
- `fx`: `gizlang_native.Library`
- `image`: `gizlang_native.Library`
- `handle`: `u64`
- `scratch`: `u64`
- `src`: `u64`
- `dst`: `u64`

### `Enhancer`

```gizlang
type Enhancer
```

RTX Video before NVENC: each source frame written into `input` (on the GPU: an RGBA image of the source's width x
height, rows of 4 width bytes) and pushed goes through the effect `e` to an output frame of width x height
(enhancedSize's), to NV12 and to the encoder, all on the GPU's stream. `close` gives the track.

- `width`: `i64`
- `height`: `i64`
- `input`: `M10.Buffer`

### `Interpolator`

```gizlang
type Interpolator
```

An interpolation before NVENC: each source frame written into `input` (on the GPU: an RGBA image of width x height,
rows of 4 width bytes) and pushed goes to the encoder after `times` - 1 frames made between it and the frame before,
so n frames give (n - 1) times + 1. `close` gives the track.

- `width`: `i64`
- `height`: `i64`
- `times`: `i64`
- `input`: `M10.Buffer`

### `Wave`

```gizlang
type Wave = { rate: i64; samples: Tensor; }
```

Audio samples: `channels` rows of the same length ([channels, n] f32 on the host) at `rate` a second.

- `rate`: `i64`
- `samples`: `M32.Tensor`

### `Pose`

```gizlang
type Pose = { detector: Onnx.Network; keypoints: Onnx.Network; gpu: Gpu; }
```

DWPose's networks on a GPU: YOLOX-L finds people (yolox_l.onnx), RTMPose-L their 133 whole-body keypoints
(dw-ll_ucoco_384.onnx).

- `detector`: `M29.Network`
- `keypoints`: `M29.Network`
- `gpu`: `M10.Gpu`

## Functions

### `h264`

```gizlang
function h264(sys: System, frames: Tensor, fps: i64, quality: i64, ffmpeg: string = "ffmpeg"): Track throws string
```

The H.264 track of `frames` ([3, n, height, width] RGB in [0, 1], f32 on a GPU; height and width even) at `fps`
frames a second: High profile, P frames only, a key frame each 250 frames, at constant quality `quality` (0 to 51,
lower is better; 22 gives 39 dB and 1.5 MB on 5 s of H3's 480p video). On NVIDIA NVENC encodes (about 2 ms a frame);
on another GPU (Apple's, AMD's), and on an NVIDIA GPU whose NVENC does not open (rented containers that hide the
encoder), `ffmpeg` (its name on PATH, or a path) does, with libx264.

Effects: fail, read, write, console, io.

### `canEnhance`

```gizlang
function canEnhance(sys: System): bool
```

Whether this process finds the RTX Video libraries (asked before a job spends the GPU on what it would enhance).

Effects: fail, io.

### `enhancedSize`

```gizlang
function enhancedSize(e: Enhance, width: i64, height: i64): Vec<i64> throws string
```

The output width and height of `e` on a video of width x height; an error names a choice that is not one, or an
output NVENC cannot take.

Effects: fail.

### `enhancer`

```gizlang
function enhancer(sys: System, gpu: Gpu, e: Enhance, width: i64, height: i64, fps: i64, quality: i64, codec: string): Enhancer throws string
```

An enhancer of `e` from width x height frames into a `codec` track (codecs(); NVENC at constant quality `quality`,
as encoder) at `fps`.

Effects: fail, read, write, console, io.

### `enhancedFrames`

```gizlang
function enhancedFrames(sys: System, frames: Tensor, e: Enhance, fps: i64, quality: i64): Track throws string
```

The `e` enhanced track (as enhancer: H.264) of `frames` ([3, n, height, width] RGB in [0, 1], f32 on a CUDA GPU, as
h264 takes them).

Effects: fail, read, write, console, io.

### `interpolationQualities`

```gizlang
function interpolationQualities(): Vec<string>
```

The qualities of an interpolation, as giz's interpolate-video takes them: the flow's grid (pixels a vector a side)
and the accelerator's effort for each.

Effects: fail.

### `canInterpolate`

```gizlang
function canInterpolate(sys: System): bool
```

Whether this host has the optical flow accelerator's library.

Effects: fail, io.

### `interpolator`

```gizlang
function interpolator(sys: System, gpu: Gpu, width: i64, height: i64, fps: i64, times: i64, effort: string, quality: i64, codec: string): Interpolator throws string
```

An interpolator of width x height frames at `fps` into a `codec` track at `fps` x `times` (2 to 8; NVENC at constant
quality `quality`, as encoder), its flow of `effort` (interpolationQualities). An error says when the driver has no
optical flow accelerator (libnvidia-opticalflow.so.1) or the GPU has none.

Effects: fail, read, write, console, io.

### `aac`

```gizlang
function aac(wave: Tensor, rate: i64, bitrate: i64): Aac throws string
```

The AAC-LC track of `wave` ([channels, samples] in [-1, 1]; 1 or 2 channels) at `rate` Hz (32000, 44100 or 48000) and about
`bitrate` bits a second at most. Each frame takes the least quantization noise, as a share of each band's power,
that fits its share of the bit rate (45 dB below at best); each channel is coded alone, with long windows.

Effects: fail, read, write, console, io.

### `wav`

```gizlang
function wav(data: bytes): Wave throws string
```

The samples of a WAV file (RIFF WAVE: PCM of 8, 16, 24 or 32 bits, or IEEE floats of 32 or 64 bits, also in
WAVE_FORMAT_EXTENSIBLE), as floats in [-1, 1) the way libsndfile reads them (a signed integer over 2^(bits - 1); 8
bits are unsigned around 128).

Effects: fail.

### `resampled`

```gizlang
function resampled(wave: Wave, to: i64): Wave
```

`wave` at rate `to`, as torchaudio's resample makes it (a windowed sinc of 6 zero crossings, Hann window, rolloff
0.99): each channel padded with zeros, then each output phase the filter of its offset.

Effects: fail.

### `audio`

```gizlang
function audio(sys: System, ffmpeg: string, path: string, most: f64 = 0.0): Wave throws string
```

The samples of the audio in the file `path` (any audio or video file ffmpeg reads: MP3, AAC, M4A, FLAC, Ogg, MP4,
WebM, ...), as the file has them: its sample rate and channels (WAV is read here, at its own rate, without ffmpeg).
`ffmpeg` is the program (its name on PATH, or a path). With `most` seconds (when above 0)
at most is read: a long file stays small.

Effects: fail, read, write, io.

### `probe`

```gizlang
function probe(sys: System, ffmpeg: string, path: string): Result<Vec<i64>, string>
```

The width, height, (whole) frame rate and frames (by its duration; 0: not known) of the first video stream of the
file `path`, by ffprobe next to `ffmpeg`.

Effects: fail, read, write, io.

### `frameIndices`

```gizlang
function frameIndices(rate: i64, target: i64, count: i64): Vec<i64>
```

The source frames that WanGP picks for `count` frames at `target` fps from a video of `rate` fps (its
_resample_frame_indices from frame 0, which H3 and Qwen's video readers share).

Effects: fail.

### `frames`

```gizlang
function frames(sys: System, ffmpeg: string, path: string, wanted: Vec<i64>, width: i64, height: i64, mask: bool): Result<Vec<bytes>, string>
```

The frames `wanted` (ascending indices, as frameIndices gives them) of the video `path`, as `width` x `height` RGB
bytes each (a mask: one byte each, 1 where the gray is over 127.5), resized by Lanczos as WanGP's
preprocess_video_with_mask does; those past its end are left out.

Effects: fail, read, write, io.

### `pose`

```gizlang
function pose(sys: System, folder: string, gpu: Gpu): Pose throws string
```

The pose networks of `folder` (yolox_l.onnx, dw-ll_ucoco_384.onnx) on `gpu`.

Effects: fail, read, write, console, io.

### `poseFrames`

```gizlang
function poseFrames(p: Pose, frames: Tensor, resolution: i64 = 512): Tensor throws string
```

The pose frames of `frames` ([3, n, height, width] RGB in [0, 1], f32): each frame's people (body, hands and face)
drawn in OpenPose's colors on black, its shorter side scaled to `resolution`, as ComfyUI's DWPreprocessor draws
them (comfyui_controlnet_aux: detection at the frame's size, a person a score of 0.3 or more, a keypoint 0.3).

Effects: fail, read, write, console, io.

## Methods

### `Enhancer.push`

```gizlang
function Enhancer.push(self: mut Enhancer): void throws string
```

The frame in `input` through the effect and the encoder.

Effects: fail, read, write, io.

### `Enhancer.close`

```gizlang
function Enhancer.close(self: mut Enhancer): Track
```

Frees the effect and the encoder: the track of the frames pushed.

Effects: fail, read, write, io.

### `Interpolator.push`

```gizlang
function Interpolator.push(self: mut Interpolator): void throws string
```

The frame in `input` after the frames made between it and the frame before, through the encoder; then `input` is
the other frame buffer.

Effects: fail, read, write, io.

### `Interpolator.close`

```gizlang
function Interpolator.close(self: mut Interpolator): Track
```

Frees the sessions and the encoder: the track of the frames pushed.

Effects: fail, read, write, io.

gizml:video

video on top of gizhw:codec and gizlang:mp4. H.264 tracks of frame tensors (NVENC, or x264 through ffmpeg on other GPUs), NVIDIA's RTX Video effects and frame interpolation, DWPose's pose frames, an AAC-LC encoder on the host, WAV files and resampling, and any audio file and the frames of any video file through ffmpeg.

gizlang

import * as Video from "gizml:video";

Types

`Enhance`

gizlang

type Enhance = { effect: string; size: string; quality: string; scale: i64; }

What the enhancement does, as giz's GizRTXVideoEnhance node takes it. effect: SUPER_RESOLUTION (for compressed video), HIGH_BITRATE (for clean video), DENOISE, DEBLUR or BICUBIC (no AI); size: HD, FHD, 2k or 4k (the long side 1280, 1920, 2560 or 3840, never below the source's); quality: LOW, MEDIUM, HIGH or ULTRA; scale: 1 to 4 times the source's sides in place of size (0: by size). DENOISE and DEBLUR keep the source's size.

effect: string
size: string
quality: string
scale: i64

`Effect`

gizlang

type Effect = { nat: Native; fx: Library; image: Library; handle: u64; scratch: u64; src: u64; dst: u64; }

The effect's libraries, an effect made and loaded, and its two images (NvCVImage, 64 bytes each) on GPU buffers.

nat: gizlang_native.Native
fx: gizlang_native.Library
image: gizlang_native.Library
handle: u64
scratch: u64
src: u64
dst: u64

`Enhancer`

gizlang

type Enhancer

RTX Video before NVENC: each source frame written into input (on the GPU: an RGBA image of the source's width x height, rows of 4 width bytes) and pushed goes through the effect e to an output frame of width x height (enhancedSize's), to NV12 and to the encoder, all on the GPU's stream. close gives the track.

width: i64
height: i64
input: M10.Buffer

`Interpolator`

gizlang

type Interpolator

An interpolation before NVENC: each source frame written into input (on the GPU: an RGBA image of width x height, rows of 4 width bytes) and pushed goes to the encoder after times - 1 frames made between it and the frame before, so n frames give (n - 1) times + 1. close gives the track.

width: i64
height: i64
times: i64
input: M10.Buffer

`Wave`

gizlang

type Wave = { rate: i64; samples: Tensor; }

Audio samples: channels rows of the same length ([channels, n] f32 on the host) at rate a second.

rate: i64
samples: M32.Tensor

`Pose`

gizlang

type Pose = { detector: Onnx.Network; keypoints: Onnx.Network; gpu: Gpu; }

DWPose's networks on a GPU: YOLOX-L finds people (yolox_l.onnx), RTMPose-L their 133 whole-body keypoints (dw-ll_ucoco_384.onnx).

detector: M29.Network
keypoints: M29.Network
gpu: M10.Gpu

Functions

`h264`

gizlang

function h264(sys: System, frames: Tensor, fps: i64, quality: i64, ffmpeg: string = "ffmpeg"): Track throws string

The H.264 track of frames ([3, n, height, width] RGB in [0, 1], f32 on a GPU; height and width even) at fps frames a second: High profile, P frames only, a key frame each 250 frames, at constant quality quality (0 to 51, lower is better; 22 gives 39 dB and 1.5 MB on 5 s of H3's 480p video). On NVIDIA NVENC encodes (about 2 ms a frame); on another GPU (Apple's, AMD's), and on an NVIDIA GPU whose NVENC does not open (rented containers that hide the encoder), ffmpeg (its name on PATH, or a path) does, with libx264.

Effects: fail, read, write, console, io.

`canEnhance`

gizlang

function canEnhance(sys: System): bool

Whether this process finds the RTX Video libraries (asked before a job spends the GPU on what it would enhance).

Effects: fail, io.

`enhancedSize`

gizlang

function enhancedSize(e: Enhance, width: i64, height: i64): Vec<i64> throws string

The output width and height of e on a video of width x height; an error names a choice that is not one, or an output NVENC cannot take.

Effects: fail.

`enhancer`

gizlang

function enhancer(sys: System, gpu: Gpu, e: Enhance, width: i64, height: i64, fps: i64, quality: i64, codec: string): Enhancer throws string

An enhancer of e from width x height frames into a codec track (codecs(); NVENC at constant quality quality, as encoder) at fps.

Effects: fail, read, write, console, io.

`enhancedFrames`

gizlang

function enhancedFrames(sys: System, frames: Tensor, e: Enhance, fps: i64, quality: i64): Track throws string

The e enhanced track (as enhancer: H.264) of frames ([3, n, height, width] RGB in [0, 1], f32 on a CUDA GPU, as h264 takes them).

Effects: fail, read, write, console, io.

`interpolationQualities`

gizlang

function interpolationQualities(): Vec<string>

The qualities of an interpolation, as giz's interpolate-video takes them: the flow's grid (pixels a vector a side) and the accelerator's effort for each.

Effects: fail.

`canInterpolate`

gizlang

function canInterpolate(sys: System): bool

Whether this host has the optical flow accelerator's library.

Effects: fail, io.

`interpolator`

gizlang

function interpolator(sys: System, gpu: Gpu, width: i64, height: i64, fps: i64, times: i64, effort: string, quality: i64, codec: string): Interpolator throws string

An interpolator of width x height frames at fps into a codec track at fps x times (2 to 8; NVENC at constant quality quality, as encoder), its flow of effort (interpolationQualities). An error says when the driver has no optical flow accelerator (libnvidia-opticalflow.so.1) or the GPU has none.

Effects: fail, read, write, console, io.

`aac`

gizlang

function aac(wave: Tensor, rate: i64, bitrate: i64): Aac throws string

The AAC-LC track of wave ([channels, samples] in [-1, 1]; 1 or 2 channels) at rate Hz (32000, 44100 or 48000) and about bitrate bits a second at most. Each frame takes the least quantization noise, as a share of each band's power, that fits its share of the bit rate (45 dB below at best); each channel is coded alone, with long windows.

Effects: fail, read, write, console, io.

`wav`

gizlang

function wav(data: bytes): Wave throws string

The samples of a WAV file (RIFF WAVE: PCM of 8, 16, 24 or 32 bits, or IEEE floats of 32 or 64 bits, also in WAVE_FORMAT_EXTENSIBLE), as floats in [-1, 1) the way libsndfile reads them (a signed integer over 2^(bits - 1); 8 bits are unsigned around 128).

Effects: fail.

`resampled`

gizlang

function resampled(wave: Wave, to: i64): Wave

wave at rate to, as torchaudio's resample makes it (a windowed sinc of 6 zero crossings, Hann window, rolloff 0.99): each channel padded with zeros, then each output phase the filter of its offset.

Effects: fail.

`audio`

gizlang

function audio(sys: System, ffmpeg: string, path: string, most: f64 = 0.0): Wave throws string

The samples of the audio in the file path (any audio or video file ffmpeg reads: MP3, AAC, M4A, FLAC, Ogg, MP4, WebM, ...), as the file has them: its sample rate and channels (WAV is read here, at its own rate, without ffmpeg). ffmpeg is the program (its name on PATH, or a path). With most seconds (when above 0) at most is read: a long file stays small.

Effects: fail, read, write, io.

`probe`

gizlang

function probe(sys: System, ffmpeg: string, path: string): Result<Vec<i64>, string>

The width, height, (whole) frame rate and frames (by its duration; 0: not known) of the first video stream of the file path, by ffprobe next to ffmpeg.

Effects: fail, read, write, io.

`frameIndices`

gizlang

function frameIndices(rate: i64, target: i64, count: i64): Vec<i64>

The source frames that WanGP picks for count frames at target fps from a video of rate fps (its _resample_frame_indices from frame 0, which H3 and Qwen's video readers share).

Effects: fail.

`frames`

gizlang

function frames(sys: System, ffmpeg: string, path: string, wanted: Vec<i64>, width: i64, height: i64, mask: bool): Result<Vec<bytes>, string>

The frames wanted (ascending indices, as frameIndices gives them) of the video path, as width x height RGB bytes each (a mask: one byte each, 1 where the gray is over 127.5), resized by Lanczos as WanGP's preprocess_video_with_mask does; those past its end are left out.

Effects: fail, read, write, io.

`pose`

gizlang

function pose(sys: System, folder: string, gpu: Gpu): Pose throws string

The pose networks of folder (yolox_l.onnx, dw-ll_ucoco_384.onnx) on gpu.

Effects: fail, read, write, console, io.

`poseFrames`

gizlang

function poseFrames(p: Pose, frames: Tensor, resolution: i64 = 512): Tensor throws string

The pose frames of frames ([3, n, height, width] RGB in [0, 1], f32): each frame's people (body, hands and face) drawn in OpenPose's colors on black, its shorter side scaled to resolution, as ComfyUI's DWPreprocessor draws them (comfyui_controlnet_aux: detection at the frame's size, a person a score of 0.3 or more, a keypoint 0.3).

Effects: fail, read, write, console, io.

Methods

`Enhancer.push`

gizlang

function Enhancer.push(self: mut Enhancer): void throws string

The frame in input through the effect and the encoder.

Effects: fail, read, write, io.

`Enhancer.close`

gizlang

function Enhancer.close(self: mut Enhancer): Track

Frees the effect and the encoder: the track of the frames pushed.

Effects: fail, read, write, io.

`Interpolator.push`

gizlang

function Interpolator.push(self: mut Interpolator): void throws string

The frame in input after the frames made between it and the frame before, through the encoder; then input is the other frame buffer.

Effects: fail, read, write, io.

`Interpolator.close`

gizlang

function Interpolator.close(self: mut Interpolator): Track

Frees the sessions and the encoder: the track of the frames pushed.

Effects: fail, read, write, io.