GizStackGizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/models/nemotron/model

Nemotron 3.5 ASR Streaming 0.6B (nvidia/nemotron-3.5-asr-streaming-0.6b), speech to text in 40 languages, as transformers' Nemotron3_5AsrForRNNT runs a whole recording: the encoder's linears on the GPU when there is one, the rest on the CPU.

The log-mel features (gizai/models/nemotron/features) go through a causal subsampling of 8 (a 3x3 convolution at stride 2 to 256 channels, then two depthwise and pointwise stages, ReLU after each) and a linear to 1024 values a frame (80 ms). 24 FastConformer blocks follow: half a feed-forward, self-attention with relative positions (Transformer-XL's, with the content and position biases u and v) over chunks of 4 frames that see 14 chunks back, a convolution module (GLU, a causal depthwise convolution of 9 taps, layer norm, SiLU), half a feed-forward and a layer norm. A language prompt (a one-hot of 128) joins each frame through a two-layer projector, and a linear takes the frame to 640 values. The RNN-T decoder is an embedding and a 2-layer LSTM of 640; the joint adds a frame and the decoder's output, takes ReLU and a linear to the 13088 tokens (the last is blank). Greedy decoding emits up to 10 tokens a frame and moves to the next frame on blank.

gizlang
import * as Model from "gizai/models/nemotron/model";

Types

Asr

gizlang
type Asr = { named: Map<string, Tensor>; mel: Vec<f64>; tokenizer: Tokenizer; languages: Vec<string>; prompts: Vec<i64>; auto: i64; }

The model on the CPU: its tensors by name, the mel filters, the tokenizer and the language prompts.

  • named: _gizlang_sdk_collections.Map<string, _gizlang_lib_ai__tensor__storage.Tensor<f32>>
  • mel: Vec<f64>
  • tokenizer: M19.Tokenizer
  • languages: Vec<string>
  • prompts: Vec<i64>
  • auto: i64

Listening

gizlang
type Listening = { tail: Vec<f32>; at: i64; total: i64; mel: Vec<f32>; prompt: i64; decoder: Decoder; done: i64; ids: Vec<i64>; }

A recording heard as it comes: its samples from at on (those the unsettled mel frames read), how many came in all, its settled mel frames, its language prompt, the decoder's state, the encoder frames decoded and their tokens.

  • tail: Vec<f32>
  • at: i64
  • total: i64
  • mel: Vec<f32>
  • prompt: i64
  • decoder: M83.Decoder
  • done: i64
  • ids: Vec<i64>

Functions

load

gizlang
function load(folder: string, gpu: Option<G.Gpu>): Asr throws string

The model of folder (model.safetensors, tokenizer.json, processor_config.json).

Effects: fail, read, write, console, io, alloc.

transcriber

gizlang
function transcriber(m: Asr): Transcriber

The speech-to-text engine of m (gizai/tasks/transcription).

Effects: fail, alloc.

service

gizlang
function service(folder: string, place: Place): Service

The service of the model of folder as giz's worker at place on gizai/host: audio.transcribe jobs, one at a time, and speech heard as it is said on gizai/tasks/transcription's route, on the GPU where there is one.

Effects: fail, alloc.

Methods

Asr.prompt

gizlang
function Asr.prompt(self: Asr, language: string): Option<i64>

The prompt of a language as the processor names it ("en-US", "de", "auto"), if the model has it.

Effects: fail.

Asr.transcribe

gizlang
function Asr.transcribe(self: Asr, samples: Vec<f32>, language: string): string throws string

The text spoken in samples (16 kHz, mono) in language ("" or "auto": the model finds it).

Effects: fail, read, write, console, io, alloc.

Asr.listening

gizlang
function Asr.listening(self: Asr, language: string): Listening throws string

A recording to hear as it comes, in language ("" or "auto": the model finds it).

Effects: fail, read, write, console, io, alloc.

Asr.heard

gizlang
function Asr.heard(self: Asr, l: mut Listening, samples: Vec<f32>, last: bool): string

The text heard in l after its next samples (last: the recording ends with them). The encoder is causal: a frame reads mel frames up to 8 times its index and attends within its chunk of CHUNK frames and the chunks before, so the frames of each whole chunk of settled mel frames are those of the whole recording, and their tokens are final; the rest wait for more samples, or for the end.

Effects: fail, read, write, console, io, alloc.

Asr.encoded

gizlang
function Asr.encoded(self: Asr, mel: Vec<f32>, prompt: i64): Tensor

The frames of the encoder and the prompt projector for the decoder of the mel frames mel (rows of MELS): [T, 640].

Effects: fail, read, write, console, io, alloc.

Constants

MODEL

gizlang
const MODEL = "nemotron-asr"

The model's name in the catalog and as giz's worker serves it.

REPO

gizlang
const REPO = "nvidia/nemotron-3.5-asr-streaming-0.6b"

Its Hugging Face repository.