# gizai:flashhead

SoulX FlashHead lite, a talking head from one picture and speech, as its reference package and giz's worker run it.

The picture is cut around its face (MediaPipe's detector, gizai:video) and scaled to 512 x 512; LTX-Video 0.9's VAE (gizai:ltx) codes it, repeated over 33 frames, to a reference latent of 5 frames of 16 x 16. The speech goes in slices of 24 frames (15360 samples at 16 kHz). For each slice, the last 8 s of speech go through wav2vec 2.0 base (gizai:wav2vec), its 12 layers' states taken at 200 frames, and 5 states around each of the slice's 33 frames make 5 latent frames of 32 audio tokens. A Wan-style transformer (30 blocks of 1536, 12 heads, its rotary over frame, row and column) predicts the flow of the noisy latent, whose first latent frames are the motion of the slice before, in 4 steps; the VAE decodes 33 frames, their colors match the picture's (Lab means and deviations), and the last 9 are coded again as the next slice's motion. Each slice gives its last 24 frames.

```gizlang
import * as Flashhead from "gizai:flashhead";
```

## Types

### `FlashHead`

```gizlang
type FlashHead = { gpu: Gpu; blocks: Vec<Block>; patch: N.Linear; head: N.Linear; headMods: Vec<Tensor>; proj1: N.Linear; proj1vf: N.Linear; proj2: N.Linear; proj3: N.Linear; projNorm: Vec<Tensor>; cos: Tensor; sin: Tensor; ones: Tensor; zeros: Tensor; map: Tensor; w2v: W2.Encoder; vae: L.VideoVae; encoder: L.VideoEncoder09; faces: V.Faces; work: Scratch; }
```

The model where it runs.

- `gpu`: `M10.Gpu`
- `blocks`: `Vec<M23.Block>`
- `patch`: `M33.Linear`
- `head`: `M33.Linear`
- `headMods`: `Vec<M40.Tensor>` — At each step: the output norm's shift and scale ([2, 1536] F32).
- `proj1`: `M33.Linear`
- `proj1vf`: `M33.Linear`
- `proj2`: `M33.Linear`
- `proj3`: `M33.Linear`
- `projNorm`: `Vec<M40.Tensor>`
- `cos`: `M40.Tensor` — The rotary's angles at each row, [1280, 768] F32 (pair d of a head at column h 64 + d).
- `sin`: `M40.Tensor`
- `ones`: `M40.Tensor`
- `zeros`: `M40.Tensor`
- `map`: `M40.Tensor`
- `w2v`: `M45.Encoder`
- `vae`: `M30.VideoVae`
- `encoder`: `M30.VideoEncoder09`
- `faces`: `M44.Faces`
- `work`: `M23.Scratch`

### `Settings`

```gizlang
type Settings = { seed: i64; motion: f64; }
```

Settings of a job: the seed, the motion scale (1: the model's own; less keeps the face nearer the picture).

- `seed`: `i64`
- `motion`: `f64`

## Functions

### `load`

```gizlang
function load(sys: System, folder: string, gpu: Gpu): FlashHead throws string
```

FlashHead lite of `folder` (Model_Lite, VAE_LTX, wav2vec2-base-960h, face_detection_full_range_sparse.tflite) on
`gpu`.

Effects: fail, read, write, console, io.

### `speechOf`

```gizlang
function speechOf(sys: System, ffmpeg: string, path: string): Vec<f32> throws W.Failure
```

The speech of a file at 16 kHz, its channels averaged (ffmpeg decodes it).

Effects: fail, read, write, io.

### `serve`

```gizlang
function serve(sys: System, folder: string, place: Hosting.Place): i64 throws string
```

Serves FlashHead lite of `folder` as giz's worker at `place`: video.talking-head jobs. The kernels build and tune on
one slice of silence at the load, not in the first request (8 s more).

Effects: fail, read, write, console, io.

## Methods

### `FlashHead.picture`

```gizlang
function FlashHead.picture(self: FlashHead, image: Image.Image, crop: bool, ratio: f64): Vec<f32> throws string
```

The picture the model takes, as the reference makes it: RGB planes ([3, 512, 512]) of bytes as F32 in [0, 1].
With `crop`, the square around the first face (its width times `ratio`, centered across, its top half a side's
0.55 above the face's center, moved inside the picture), scaled by Pillow's BICUBIC; then (or without) scaled by
Pillow's BILINEAR to cover 512 x 512 and cut at the center.

Effects: fail, read, write, console, io.

### `FlashHead.talk`

```gizlang
function FlashHead.talk(self: FlashHead, picture: Vec<f32>, speech: Vec<f32>, s: Settings, out: mut V.H264, total: i64, alive: (step: i64, total: i64) => bool): void throws string
```

The frames of a talking head of `picture` (FlashHead.picture's planes) saying `speech` (16 kHz mono), pushed to
`out` slice by slice until it has `total`; `alive` false stops.

Effects: fail, read, write, console, io.

### `FlashHead.video`

```gizlang
function FlashHead.video(self: FlashHead, sys: System, ffmpeg: string, picture: Vec<f32>, speech: Vec<f32>, s: Settings, alive: (step: i64, total: i64) => bool): bytes throws string
```

An MP4 of the talking head of `picture` saying `speech` (16 kHz mono): H.264 at 25 fps, a frame for each 1 / 25 s
of speech, and the speech as AAC.

Effects: fail, read, write, console, io.

## Constants

### `MODEL`

```gizlang
const MODEL = "soulx-flashhead-lite"
```

The model id and operation of giz's worker.

### `OPERATION`

```gizlang
const OPERATION = "video.talking-head"
```

### `FPS`

```gizlang
const FPS = 25
```

The frames a second, and the speech's samples a second.

### `RATE`

```gizlang
const RATE = 16000
```