# gizml:video video on top of gizhw:codec and gizlang:mp4. H.264 tracks of frame tensors (NVENC, or x264 through ffmpeg on other GPUs), NVIDIA's RTX Video effects and frame interpolation, DWPose's pose frames, an AAC-LC encoder on the host, WAV files and resampling, and any audio file and the frames of any video file through ffmpeg. ```gizlang import * as Video from "gizml:video"; ``` ## Types ### `Enhance` ```gizlang type Enhance = { effect: string; size: string; quality: string; scale: i64; } ``` What the enhancement does, as giz's GizRTXVideoEnhance node takes it. `effect`: SUPER_RESOLUTION (for compressed video), HIGH_BITRATE (for clean video), DENOISE, DEBLUR or BICUBIC (no AI); `size`: HD, FHD, 2k or 4k (the long side 1280, 1920, 2560 or 3840, never below the source's); `quality`: LOW, MEDIUM, HIGH or ULTRA; `scale`: 1 to 4 times the source's sides in place of `size` (0: by size). DENOISE and DEBLUR keep the source's size. - `effect`: `string` - `size`: `string` - `quality`: `string` - `scale`: `i64` ### `Effect` ```gizlang type Effect = { nat: Native; fx: Library; image: Library; handle: u64; scratch: u64; src: u64; dst: u64; } ``` The effect's libraries, an effect made and loaded, and its two images (NvCVImage, 64 bytes each) on GPU buffers. - `nat`: `gizlang_native.Native` - `fx`: `gizlang_native.Library` - `image`: `gizlang_native.Library` - `handle`: `u64` - `scratch`: `u64` - `src`: `u64` - `dst`: `u64` ### `Enhancer` ```gizlang type Enhancer ``` RTX Video before NVENC: each source frame written into `input` (on the GPU: an RGBA image of the source's width x height, rows of 4 width bytes) and pushed goes through the effect `e` to an output frame of width x height (enhancedSize's), to NV12 and to the encoder, all on the GPU's stream. `close` gives the track. - `width`: `i64` - `height`: `i64` - `input`: `M10.Buffer` ### `Interpolator` ```gizlang type Interpolator ``` An interpolation before NVENC: each source frame written into `input` (on the GPU: an RGBA image of width x height, rows of 4 width bytes) and pushed goes to the encoder after `times` - 1 frames made between it and the frame before, so n frames give (n - 1) times + 1. `close` gives the track. - `width`: `i64` - `height`: `i64` - `times`: `i64` - `input`: `M10.Buffer` ### `Wave` ```gizlang type Wave = { rate: i64; samples: Tensor; } ``` Audio samples: `channels` rows of the same length ([channels, n] f32 on the host) at `rate` a second. - `rate`: `i64` - `samples`: `M32.Tensor` ### `Pose` ```gizlang type Pose = { detector: Onnx.Network; keypoints: Onnx.Network; gpu: Gpu; } ``` DWPose's networks on a GPU: YOLOX-L finds people (yolox_l.onnx), RTMPose-L their 133 whole-body keypoints (dw-ll_ucoco_384.onnx). - `detector`: `M29.Network` - `keypoints`: `M29.Network` - `gpu`: `M10.Gpu` ## Functions ### `h264` ```gizlang function h264(sys: System, frames: Tensor, fps: i64, quality: i64, ffmpeg: string = "ffmpeg"): Track throws string ``` The H.264 track of `frames` ([3, n, height, width] RGB in [0, 1], f32 on a GPU; height and width even) at `fps` frames a second: High profile, P frames only, a key frame each 250 frames, at constant quality `quality` (0 to 51, lower is better; 22 gives 39 dB and 1.5 MB on 5 s of H3's 480p video). On NVIDIA NVENC encodes (about 2 ms a frame); on another GPU (Apple's, AMD's), and on an NVIDIA GPU whose NVENC does not open (rented containers that hide the encoder), `ffmpeg` (its name on PATH, or a path) does, with libx264. Effects: fail, read, write, console, io. ### `canEnhance` ```gizlang function canEnhance(sys: System): bool ``` Whether this process finds the RTX Video libraries (asked before a job spends the GPU on what it would enhance). Effects: fail, io. ### `enhancedSize` ```gizlang function enhancedSize(e: Enhance, width: i64, height: i64): Vec throws string ``` The output width and height of `e` on a video of width x height; an error names a choice that is not one, or an output NVENC cannot take. Effects: fail. ### `enhancer` ```gizlang function enhancer(sys: System, gpu: Gpu, e: Enhance, width: i64, height: i64, fps: i64, quality: i64, codec: string): Enhancer throws string ``` An enhancer of `e` from width x height frames into a `codec` track (codecs(); NVENC at constant quality `quality`, as encoder) at `fps`. Effects: fail, read, write, console, io. ### `enhancedFrames` ```gizlang function enhancedFrames(sys: System, frames: Tensor, e: Enhance, fps: i64, quality: i64): Track throws string ``` The `e` enhanced track (as enhancer: H.264) of `frames` ([3, n, height, width] RGB in [0, 1], f32 on a CUDA GPU, as h264 takes them). Effects: fail, read, write, console, io. ### `interpolationQualities` ```gizlang function interpolationQualities(): Vec ``` The qualities of an interpolation, as giz's interpolate-video takes them: the flow's grid (pixels a vector a side) and the accelerator's effort for each. Effects: fail. ### `canInterpolate` ```gizlang function canInterpolate(sys: System): bool ``` Whether this host has the optical flow accelerator's library. Effects: fail, io. ### `interpolator` ```gizlang function interpolator(sys: System, gpu: Gpu, width: i64, height: i64, fps: i64, times: i64, effort: string, quality: i64, codec: string): Interpolator throws string ``` An interpolator of width x height frames at `fps` into a `codec` track at `fps` x `times` (2 to 8; NVENC at constant quality `quality`, as encoder), its flow of `effort` (interpolationQualities). An error says when the driver has no optical flow accelerator (libnvidia-opticalflow.so.1) or the GPU has none. Effects: fail, read, write, console, io. ### `aac` ```gizlang function aac(wave: Tensor, rate: i64, bitrate: i64): Aac throws string ``` The AAC-LC track of `wave` ([channels, samples] in [-1, 1]; 1 or 2 channels) at `rate` Hz (32000, 44100 or 48000) and about `bitrate` bits a second at most. Each frame takes the least quantization noise, as a share of each band's power, that fits its share of the bit rate (45 dB below at best); each channel is coded alone, with long windows. Effects: fail, read, write, console, io. ### `wav` ```gizlang function wav(data: bytes): Wave throws string ``` The samples of a WAV file (RIFF WAVE: PCM of 8, 16, 24 or 32 bits, or IEEE floats of 32 or 64 bits, also in WAVE_FORMAT_EXTENSIBLE), as floats in [-1, 1) the way libsndfile reads them (a signed integer over 2^(bits - 1); 8 bits are unsigned around 128). Effects: fail. ### `resampled` ```gizlang function resampled(wave: Wave, to: i64): Wave ``` `wave` at rate `to`, as torchaudio's resample makes it (a windowed sinc of 6 zero crossings, Hann window, rolloff 0.99): each channel padded with zeros, then each output phase the filter of its offset. Effects: fail. ### `audio` ```gizlang function audio(sys: System, ffmpeg: string, path: string, most: f64 = 0.0): Wave throws string ``` The samples of the audio in the file `path` (any audio or video file ffmpeg reads: MP3, AAC, M4A, FLAC, Ogg, MP4, WebM, ...), as the file has them: its sample rate and channels (WAV is read here, at its own rate, without ffmpeg). `ffmpeg` is the program (its name on PATH, or a path). With `most` seconds (when above 0) at most is read: a long file stays small. Effects: fail, read, write, io. ### `probe` ```gizlang function probe(sys: System, ffmpeg: string, path: string): Result, string> ``` The width, height, (whole) frame rate and frames (by its duration; 0: not known) of the first video stream of the file `path`, by ffprobe next to `ffmpeg`. Effects: fail, read, write, io. ### `frameIndices` ```gizlang function frameIndices(rate: i64, target: i64, count: i64): Vec ``` The source frames that WanGP picks for `count` frames at `target` fps from a video of `rate` fps (its _resample_frame_indices from frame 0, which H3 and Qwen's video readers share). Effects: fail. ### `frames` ```gizlang function frames(sys: System, ffmpeg: string, path: string, wanted: Vec, width: i64, height: i64, mask: bool): Result, string> ``` The frames `wanted` (ascending indices, as frameIndices gives them) of the video `path`, as `width` x `height` RGB bytes each (a mask: one byte each, 1 where the gray is over 127.5), resized by Lanczos as WanGP's preprocess_video_with_mask does; those past its end are left out. Effects: fail, read, write, io. ### `pose` ```gizlang function pose(sys: System, folder: string, gpu: Gpu): Pose throws string ``` The pose networks of `folder` (yolox_l.onnx, dw-ll_ucoco_384.onnx) on `gpu`. Effects: fail, read, write, console, io. ### `poseFrames` ```gizlang function poseFrames(p: Pose, frames: Tensor, resolution: i64 = 512): Tensor throws string ``` The pose frames of `frames` ([3, n, height, width] RGB in [0, 1], f32): each frame's people (body, hands and face) drawn in OpenPose's colors on black, its shorter side scaled to `resolution`, as ComfyUI's DWPreprocessor draws them (comfyui_controlnet_aux: detection at the frame's size, a person a score of 0.3 or more, a keypoint 0.3). Effects: fail, read, write, console, io. ## Methods ### `Enhancer.push` ```gizlang function Enhancer.push(self: mut Enhancer): void throws string ``` The frame in `input` through the effect and the encoder. Effects: fail, read, write, io. ### `Enhancer.close` ```gizlang function Enhancer.close(self: mut Enhancer): Track ``` Frees the effect and the encoder: the track of the frames pushed. Effects: fail, read, write, io. ### `Interpolator.push` ```gizlang function Interpolator.push(self: mut Interpolator): void throws string ``` The frame in `input` after the frames made between it and the frame before, through the encoder; then `input` is the other frame buffer. Effects: fail, read, write, io. ### `Interpolator.close` ```gizlang function Interpolator.close(self: mut Interpolator): Track ``` Frees the sessions and the encoder: the track of the frames pushed. Effects: fail, read, write, io.