GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/models/lam/expression

speech to face motion. LAM's Audio2Expression (streaming) gives 52 ARKit blendshape weights a frame, 30 frames a second, from 16 kHz speech.

wav2vec 2.0 base's features (gizai/parts/wav2vec) are resampled from 50 to 30 a second (linear, ends aligned) before its transformer. Its last state is projected to 512 and joined with an identity's 64 values (a 1x1 convolution of one of 12 one-hot speakers). Six 3-tap convolutions follow, each with a layer norm and ReLU (the first three with a residual), then a linear to 52 and a sigmoid. The streaming model reads a window of 64 frames (2.13 s) and keeps the frames of the new speech.

gizlang
import * as Expression from "gizai/models/lam/expression";

Types

Model

gizlang
type Model = { encoder: W2.Encoder; w: Linears.Scope; gpu: Option<Gpu>; }

The model where it runs: the speech encoder, and the head's tensors by name.

  • encoder: _gizlang_lib_ai__parts__wav2vec.Encoder
  • w: _gizlang_lib_ai__netops__linear.Scope
  • gpu: _gizlang_sdk_core.Option<M7.Gpu>

Functions

load

gizlang
function load(file: string, gpu: Option<Gpu>): Model throws string

The model in file, as LAM publishes it (lam_audio2exp_streaming.tar, a torch.save checkpoint whose names start with "module.backbone.") or as safetensors, on gpu when there is one.

Effects: fail, read, write, console, io, alloc.

expressions

gizlang
function expressions(m: Model, samples: Tensor, id: i64): Tensor

The [frames, WEIGHTS] weights of samples ([n, 1] 16 kHz) for speaker id (0 to SPEAKERS - 1), frames = ceil(n / 16000 * 30). Nothing waits for the GPU, and the sizes follow n alone: a call can be recorded (Gpu.record) and replayed with new samples.

Effects: fail, read, write, console, io, alloc.

finished

gizlang
function finished(weights: Vec<f32>, samples: Vec<f32>, seed: u64): Vec<f64>

weights ([frames, WEIGHTS] host rows, as expressions gives them) made ready to drive a face, as LAM's player does: the mouth at a tenth where the speech is silent for 7 frames or more (a frame's RMS below 0.001), each weight smoothed over 5 frames (Savitzky-Golay, quadratic), and the eyes blinking every 2 to 4 s (the model's own blink weights dropped). seed picks the blinks: the same seed, the same face.

Effects: fail, read, write, console, io, alloc.

Constants

WEIGHTS

gizlang
const WEIGHTS: i64 = 52

The weights a frame: ARKit's blendshapes in ARKit's order (browDownLeft first, tongueOut last).

FPS

gizlang
const FPS: i64 = 30

The frames a second.

SPEAKERS

gizlang
const SPEAKERS: i64 = 12

The speakers the identity input takes.