# gizai/models/lam/expression

speech to face motion. LAM's Audio2Expression (streaming) gives 52 ARKit blendshape weights a frame, 30 frames a second, from 16 kHz speech.

wav2vec 2.0 base's features (gizai/parts/wav2vec) are resampled from 50 to 30 a second (linear, ends aligned) before its transformer. Its last state is projected to 512 and joined with an identity's 64 values (a 1x1 convolution of one of 12 one-hot speakers). Six 3-tap convolutions follow, each with a layer norm and ReLU (the first three with a residual), then a linear to 52 and a sigmoid. The streaming model reads a window of 64 frames (2.13 s) and keeps the frames of the new speech.

```gizlang
import * as Expression from "gizai/models/lam/expression";
```

## Types

### `Model`

```gizlang
type Model = { encoder: W2.Encoder; w: Linears.Scope; gpu: Option<Gpu>; }
```

The model where it runs: the speech encoder, and the head's tensors by name.

- `encoder`: `_gizlang_lib_ai__parts__wav2vec.Encoder`
- `w`: `_gizlang_lib_ai__netops__linear.Scope`
- `gpu`: `_gizlang_sdk_core.Option<M7.Gpu>`

## Functions

### `load`

```gizlang
function load(file: string, gpu: Option<Gpu>): Model throws string
```

The model in `file`, as LAM publishes it (lam_audio2exp_streaming.tar, a torch.save checkpoint whose names start
with "module.backbone.") or as safetensors, on `gpu` when there is one.

Effects: fail, read, write, console, io, alloc.

### `expressions`

```gizlang
function expressions(m: Model, samples: Tensor, id: i64): Tensor
```

The [frames, WEIGHTS] weights of `samples` ([n, 1] 16 kHz) for speaker `id` (0 to SPEAKERS - 1), frames =
ceil(n / 16000 * 30). Nothing waits for the GPU, and the sizes follow n alone: a call can be recorded (Gpu.record)
and replayed with new samples.

Effects: fail, read, write, console, io, alloc.

### `finished`

```gizlang
function finished(weights: Vec<f32>, samples: Vec<f32>, seed: u64): Vec<f64>
```

`weights` ([frames, WEIGHTS] host rows, as expressions gives them) made ready to drive a face, as LAM's player
does: the mouth at a tenth where the speech is silent for 7 frames or more (a frame's RMS below 0.001), each weight
smoothed over 5 frames (Savitzky-Golay, quadratic), and the eyes blinking every 2 to 4 s (the model's own blink
weights dropped). `seed` picks the blinks: the same seed, the same face.

Effects: fail, read, write, console, io, alloc.

## Constants

### `WEIGHTS`

```gizlang
const WEIGHTS: i64 = 52
```

The weights a frame: ARKit's blendshapes in ARKit's order (browDownLeft first, tongueOut last).

### `FPS`

```gizlang
const FPS: i64 = 30
```

The frames a second.

### `SPEAKERS`

```gizlang
const SPEAKERS: i64 = 12
```

The speakers the identity input takes.

