gizai:wav2vec

the speech encoder of wav2vec 2.0 base and HuBERT base (one architecture, with transformers' names).

16 kHz samples go through seven convolutions (no bias; the first's GroupNorm has a group a channel; GELU after each) to 512 values every 320 samples (50 a second): the features. A LayerNorm and a linear take each to 768, a grouped positional convolution (16 groups, 128 taps, its weight norm applied when it loads; the extra last row of an even kernel dropped) through GELU is added, then a LayerNorm and 12 post-norm transformer layers (12 heads, GELU). The hidden states are the rows after that LayerNorm and after each layer: 13 of [frames, 768].

import * as Wav2vec from "gizai:wav2vec";

Types

Encoder

type Encoder = { gpu: Option<Gpu>; named: Map<string, Tensor>; pos: Vec<Tensor>; }

The encoder where it runs: its tensors by name without the prefix (F32), and the positional convolution's weight with its norm applied, one part a group.

• gpu: _gizlang_sdk_core.Option<M10.Gpu>
• named: _gizlang_sdk_collections.Map<string, M40.Tensor>
• pos: Vec<M40.Tensor>

Functions

load

function load(sys: System, named: Map<string, Stored>, prefix: string, gpu: Option<Gpu>): Encoder throws string

The encoder whose tensors are prefix and a transformers name in named ("wav2vec2.", "semantic_model."), on gpu when there is one. The positional weight is its g and v (weight_g, weight_v, or their parametrizations).

Effects: fail, read, write, console, io.

Methods

Encoder.features

function Encoder.features(self: Encoder, samples: Vec<f32>): Tensor

The features of the samples (16 kHz): [frames, 512] F32, where the encoder runs.

Effects: fail, read, write, console, io.

Encoder.states

function Encoder.states(self: Encoder, features: Tensor): Vec<Tensor>

The 13 hidden states ([frames, 768] F32 each, where the encoder runs) of features ([frames, 512]).

Effects: fail, read, write, console, io.

Constants

RATE

const RATE = 16000

The rate of the samples the encoder takes.

FEATURES

const FEATURES = 512

The values of a feature, and of a hidden state.

WIDTH

const WIDTH = 768

LAYERS

const LAYERS = 12

The transformer's layers: the hidden states are one more.