gizai/models/lam/expression
speech to face motion. LAM's Audio2Expression (streaming) gives 52 ARKit blendshape weights a frame, 30 frames a second, from 16 kHz speech.
wav2vec 2.0 base's features (gizai/parts/wav2vec) are resampled from 50 to 30 a second (linear, ends aligned) before its transformer. Its last state is projected to 512 and joined with an identity's 64 values (a 1x1 convolution of one of 12 one-hot speakers). Six 3-tap convolutions follow, each with a layer norm and ReLU (the first three with a residual), then a linear to 52 and a sigmoid. The streaming model reads a window of 64 frames (2.13 s) and keeps the frames of the new speech.
import * as Expression from "gizai/models/lam/expression";Types
Model
type Model = { encoder: W2.Encoder; w: Linears.Scope; gpu: Option<Gpu>; }The model where it runs: the speech encoder, and the head's tensors by name.
encoder:_gizlang_lib_ai__parts__wav2vec.Encoderw:_gizlang_lib_ai__netops__linear.Scopegpu:_gizlang_sdk_core.Option<M7.Gpu>
Functions
load
function load(file: string, gpu: Option<Gpu>): Model throws stringThe model in file, as LAM publishes it (lam_audio2exp_streaming.tar, a torch.save checkpoint whose names start
with "module.backbone.") or as safetensors, on gpu when there is one.
Effects: fail, read, write, console, io, alloc.
expressions
function expressions(m: Model, samples: Tensor, id: i64): TensorThe [frames, WEIGHTS] weights of samples ([n, 1] 16 kHz) for speaker id (0 to SPEAKERS - 1), frames =
ceil(n / 16000 * 30). Nothing waits for the GPU, and the sizes follow n alone: a call can be recorded (Gpu.record)
and replayed with new samples.
Effects: fail, read, write, console, io, alloc.
finished
function finished(weights: Vec<f32>, samples: Vec<f32>, seed: u64): Vec<f64>weights ([frames, WEIGHTS] host rows, as expressions gives them) made ready to drive a face, as LAM's player
does: the mouth at a tenth where the speech is silent for 7 frames or more (a frame's RMS below 0.001), each weight
smoothed over 5 frames (Savitzky-Golay, quadratic), and the eyes blinking every 2 to 4 s (the model's own blink
weights dropped). seed picks the blinks: the same seed, the same face.
Effects: fail, read, write, console, io, alloc.
Constants
WEIGHTS
const WEIGHTS: i64 = 52The weights a frame: ARKit's blendshapes in ARKit's order (browDownLeft first, tongueOut last).
FPS
const FPS: i64 = 30The frames a second.
SPEAKERS
const SPEAKERS: i64 = 12The speakers the identity input takes.