# gizai:flashhead

SoulX FlashHead lite, a talking head from one picture and speech, as its reference package and giz's worker run it.

The picture is cut around its face (MediaPipe's detector, gizai:video) and scaled to 512 x 512; LTX-Video 0.9's VAE (gizai:ltx) codes it, repeated over 33 frames, to a reference latent of 5 frames of 16 x 16. The speech goes in slices of 24 frames (15360 samples at 16 kHz). For each slice, the last 8 s of speech go through wav2vec 2.0 base (gizai:wav2vec), its 12 layers' states taken at 200 frames, and 5 states around each of the slice's 33 frames make 5 latent frames of 32 audio tokens. A Wan-style transformer (30 blocks of 1536, 12 heads, its rotary over frame, row and column) predicts the flow of the noisy latent, whose first latent frames are the motion of the slice before, in 4 steps; the VAE decodes 33 frames, their colors match the picture's (Lab means and deviations), and the last 9 are coded again as the next slice's motion. Each slice gives its last 24 frames.

```gizlang
import * as Flashhead from "gizai:flashhead";
```

## Types

### `FlashHead`

```gizlang
type FlashHead = { gpu: Gpu; blocks: Vec<Block>; patch: N.Linear; head: N.Linear; headMods: Vec<Tensor>; proj1: N.Linear; proj1vf: N.Linear; proj2: N.Linear; proj3: N.Linear; projNorm: Vec<Tensor>; cos: Tensor; sin: Tensor; ones: Tensor; zeros: Tensor; map: Tensor; w2v: W2.Encoder; vae: L.VideoVae; encoder: L.VideoEncoder09; faces: V.Faces; work: Scratch; }
```

The model where it runs.

- `gpu`: `M10.Gpu`
- `blocks`: `Vec<M23.Block>`
- `patch`: `M33.Linear`
- `head`: `M33.Linear`
- `headMods`: `Vec<M40.Tensor>` — At each step: the output norm's shift and scale ([2, 1536] F32).
- `proj1`: `M33.Linear`
- `proj1vf`: `M33.Linear`
- `proj2`: `M33.Linear`
- `proj3`: `M33.Linear`
- `projNorm`: `Vec<M40.Tensor>`
- `cos`: `M40.Tensor` — The rotary's angles at each row, [1280, 768] F32 (pair d of a head at column h 64 + d).
- `sin`: `M40.Tensor`
- `ones`: `M40.Tensor`
- `zeros`: `M40.Tensor`
- `map`: `M40.Tensor`
- `w2v`: `M45.Encoder`
- `vae`: `M30.VideoVae`
- `encoder`: `M30.VideoEncoder09`
- `faces`: `M44.Faces`
- `work`: `M23.Scratch`

### `Settings`

```gizlang
type Settings = { seed: i64; motion: f64; }
```

Settings of a job: the seed, the motion scale (1: the model's own; less keeps the face nearer the picture).

- `seed`: `i64`
- `motion`: `f64`

## Functions

### `load`

```gizlang
function load(sys: System, folder: string, gpu: Gpu): FlashHead throws string
```

FlashHead lite of `folder` (Model_Lite, VAE_LTX, wav2vec2-base-960h, face_detection_full_range_sparse.tflite) on
`gpu`.

Effects: fail, read, write, console, io.

### `speechOf`

```gizlang
function speechOf(sys: System, ffmpeg: string, path: string): Vec<f32> throws W.Failure
```

The speech of a file at 16 kHz, its channels averaged (ffmpeg decodes it).

Effects: fail, read, write, io.

### `serve`

```gizlang
function serve(sys: System, folder: string, place: Hosting.Place): i64 throws string
```

Serves FlashHead lite of `folder` as giz's worker at `place`: video.talking-head jobs. The kernels build and tune on
one slice of silence at the load, not in the first request (8 s more).

Effects: fail, read, write, console, io.

## Methods

### `FlashHead.picture`

```gizlang
function FlashHead.picture(self: FlashHead, image: Image.Image, crop: bool, ratio: f64): Vec<f32> throws string
```

The picture the model takes, as the reference makes it: RGB planes ([3, 512, 512]) of bytes as F32 in [0, 1].
With `crop`, the square around the first face (its width times `ratio`, centered across, its top half a side's
0.55 above the face's center, moved inside the picture), scaled by Pillow's BICUBIC; then (or without) scaled by
Pillow's BILINEAR to cover 512 x 512 and cut at the center.

Effects: fail, read, write, console, io.

### `FlashHead.talk`

```gizlang
function FlashHead.talk(self: FlashHead, picture: Vec<f32>, speech: Vec<f32>, s: Settings, out: mut V.H264, total: i64, alive: (step: i64, total: i64) => bool): void throws string
```

The frames of a talking head of `picture` (FlashHead.picture's planes) saying `speech` (16 kHz mono), pushed to
`out` slice by slice until it has `total`; `alive` false stops.

Effects: fail, read, write, console, io.

### `FlashHead.video`

```gizlang
function FlashHead.video(self: FlashHead, sys: System, ffmpeg: string, picture: Vec<f32>, speech: Vec<f32>, s: Settings, alive: (step: i64, total: i64) => bool): bytes throws string
```

An MP4 of the talking head of `picture` saying `speech` (16 kHz mono): H.264 at 25 fps, a frame for each 1 / 25 s
of speech, and the speech as AAC.

Effects: fail, read, write, console, io.

## Constants

### `MODEL`

```gizlang
const MODEL = "soulx-flashhead-lite"
```

The model id and operation of giz's worker.

### `OPERATION`

```gizlang
const OPERATION = "video.talking-head"
```

### `FPS`

```gizlang
const FPS = 25
```

The frames a second, and the speech's samples a second.

### `RATE`

```gizlang
const RATE = 16000
```

gizai:flashhead

SoulX FlashHead lite, a talking head from one picture and speech, as its reference package and giz's worker run it.

The picture is cut around its face (MediaPipe's detector, gizai:video) and scaled to 512 x 512; LTX-Video 0.9's VAE (gizai:ltx) codes it, repeated over 33 frames, to a reference latent of 5 frames of 16 x 16. The speech goes in slices of 24 frames (15360 samples at 16 kHz). For each slice, the last 8 s of speech go through wav2vec 2.0 base (gizai:wav2vec), its 12 layers' states taken at 200 frames, and 5 states around each of the slice's 33 frames make 5 latent frames of 32 audio tokens. A Wan-style transformer (30 blocks of 1536, 12 heads, its rotary over frame, row and column) predicts the flow of the noisy latent, whose first latent frames are the motion of the slice before, in 4 steps; the VAE decodes 33 frames, their colors match the picture's (Lab means and deviations), and the last 9 are coded again as the next slice's motion. Each slice gives its last 24 frames.

gizlang

import * as Flashhead from "gizai:flashhead";

Types

`FlashHead`

gizlang

type FlashHead = { gpu: Gpu; blocks: Vec<Block>; patch: N.Linear; head: N.Linear; headMods: Vec<Tensor>; proj1: N.Linear; proj1vf: N.Linear; proj2: N.Linear; proj3: N.Linear; projNorm: Vec<Tensor>; cos: Tensor; sin: Tensor; ones: Tensor; zeros: Tensor; map: Tensor; w2v: W2.Encoder; vae: L.VideoVae; encoder: L.VideoEncoder09; faces: V.Faces; work: Scratch; }

The model where it runs.

gpu: M10.Gpu
blocks: Vec<M23.Block>
patch: M33.Linear
head: M33.Linear
headMods: Vec<M40.Tensor> — At each step: the output norm's shift and scale ([2, 1536] F32).
proj1: M33.Linear
proj1vf: M33.Linear
proj2: M33.Linear
proj3: M33.Linear
projNorm: Vec<M40.Tensor>
cos: M40.Tensor — The rotary's angles at each row, [1280, 768] F32 (pair d of a head at column h 64 + d).
sin: M40.Tensor
ones: M40.Tensor
zeros: M40.Tensor
map: M40.Tensor
w2v: M45.Encoder
vae: M30.VideoVae
encoder: M30.VideoEncoder09
faces: M44.Faces
work: M23.Scratch

`Settings`

gizlang

type Settings = { seed: i64; motion: f64; }

Settings of a job: the seed, the motion scale (1: the model's own; less keeps the face nearer the picture).

seed: i64
motion: f64

Functions

`load`

gizlang

function load(sys: System, folder: string, gpu: Gpu): FlashHead throws string

FlashHead lite of folder (Model_Lite, VAE_LTX, wav2vec2-base-960h, face_detection_full_range_sparse.tflite) on gpu.

Effects: fail, read, write, console, io.

`speechOf`

gizlang

function speechOf(sys: System, ffmpeg: string, path: string): Vec<f32> throws W.Failure

The speech of a file at 16 kHz, its channels averaged (ffmpeg decodes it).

Effects: fail, read, write, io.

`serve`

gizlang

function serve(sys: System, folder: string, place: Hosting.Place): i64 throws string

Serves FlashHead lite of folder as giz's worker at place: video.talking-head jobs. The kernels build and tune on one slice of silence at the load, not in the first request (8 s more).

Effects: fail, read, write, console, io.

Methods

`FlashHead.picture`

gizlang

function FlashHead.picture(self: FlashHead, image: Image.Image, crop: bool, ratio: f64): Vec<f32> throws string

The picture the model takes, as the reference makes it: RGB planes ([3, 512, 512]) of bytes as F32 in [0, 1]. With crop, the square around the first face (its width times ratio, centered across, its top half a side's 0.55 above the face's center, moved inside the picture), scaled by Pillow's BICUBIC; then (or without) scaled by Pillow's BILINEAR to cover 512 x 512 and cut at the center.

Effects: fail, read, write, console, io.

`FlashHead.talk`

gizlang

function FlashHead.talk(self: FlashHead, picture: Vec<f32>, speech: Vec<f32>, s: Settings, out: mut V.H264, total: i64, alive: (step: i64, total: i64) => bool): void throws string

The frames of a talking head of picture (FlashHead.picture's planes) saying speech (16 kHz mono), pushed to out slice by slice until it has total; alive false stops.

Effects: fail, read, write, console, io.

`FlashHead.video`

gizlang

function FlashHead.video(self: FlashHead, sys: System, ffmpeg: string, picture: Vec<f32>, speech: Vec<f32>, s: Settings, alive: (step: i64, total: i64) => bool): bytes throws string

An MP4 of the talking head of picture saying speech (16 kHz mono): H.264 at 25 fps, a frame for each 1 / 25 s of speech, and the speech as AAC.

Effects: fail, read, write, console, io.

Constants

`MODEL`

gizlang

const MODEL = "soulx-flashhead-lite"

The model id and operation of giz's worker.

`OPERATION`

gizlang

const OPERATION = "video.talking-head"

`FPS`

gizlang

const FPS = 25

The frames a second, and the speech's samples a second.

`RATE`

gizlang

const RATE = 16000