gizai/models/nemotron/model
Nemotron 3.5 ASR Streaming 0.6B (nvidia/nemotron-3.5-asr-streaming-0.6b), speech to text in 40 languages, as transformers' Nemotron3_5AsrForRNNT runs a whole recording: the encoder's linears on the GPU when there is one, the rest on the CPU.
The log-mel features (gizai/models/nemotron/features) go through a causal subsampling of 8 (a 3x3 convolution at stride 2 to 256 channels, then two depthwise and pointwise stages, ReLU after each) and a linear to 1024 values a frame (80 ms). 24 FastConformer blocks follow: half a feed-forward, self-attention with relative positions (Transformer-XL's, with the content and position biases u and v) over chunks of 4 frames that see 14 chunks back, a convolution module (GLU, a causal depthwise convolution of 9 taps, layer norm, SiLU), half a feed-forward and a layer norm. A language prompt (a one-hot of 128) joins each frame through a two-layer projector, and a linear takes the frame to 640 values. The RNN-T decoder is an embedding and a 2-layer LSTM of 640; the joint adds a frame and the decoder's output, takes ReLU and a linear to the 13088 tokens (the last is blank). Greedy decoding emits up to 10 tokens a frame and moves to the next frame on blank.
import * as Model from "gizai/models/nemotron/model";Types
Asr
type Asr = { named: Map<string, Tensor>; mel: Vec<f64>; tokenizer: Tokenizer; languages: Vec<string>; prompts: Vec<i64>; auto: i64; }The model on the CPU: its tensors by name, the mel filters, the tokenizer and the language prompts.
named:_gizlang_sdk_collections.Map<string, _gizlang_lib_ai__tensor__storage.Tensor<f32>>mel:Vec<f64>tokenizer:M19.Tokenizerlanguages:Vec<string>prompts:Vec<i64>auto:i64
Listening
type Listening = { tail: Vec<f32>; at: i64; total: i64; mel: Vec<f32>; prompt: i64; decoder: Decoder; done: i64; ids: Vec<i64>; }A recording heard as it comes: its samples from at on (those the unsettled mel frames read), how many came in
all, its settled mel frames, its language prompt, the decoder's state, the encoder frames decoded and their
tokens.
tail:Vec<f32>at:i64total:i64mel:Vec<f32>prompt:i64decoder:M83.Decoderdone:i64ids:Vec<i64>
Functions
load
function load(folder: string, gpu: Option<G.Gpu>): Asr throws stringThe model of folder (model.safetensors, tokenizer.json, processor_config.json).
Effects: fail, read, write, console, io, alloc.
transcriber
function transcriber(m: Asr): TranscriberThe speech-to-text engine of m (gizai/tasks/transcription).
Effects: fail, alloc.
service
function service(folder: string, place: Place): ServiceThe service of the model of folder as giz's worker at place on gizai/host: audio.transcribe jobs, one at a
time, and speech heard as it is said on gizai/tasks/transcription's route, on the GPU where there is one.
Effects: fail, alloc.
Methods
Asr.prompt
function Asr.prompt(self: Asr, language: string): Option<i64>The prompt of a language as the processor names it ("en-US", "de", "auto"), if the model has it.
Effects: fail.
Asr.transcribe
function Asr.transcribe(self: Asr, samples: Vec<f32>, language: string): string throws stringThe text spoken in samples (16 kHz, mono) in language ("" or "auto": the model finds it).
Effects: fail, read, write, console, io, alloc.
Asr.listening
function Asr.listening(self: Asr, language: string): Listening throws stringA recording to hear as it comes, in language ("" or "auto": the model finds it).
Effects: fail, read, write, console, io, alloc.
Asr.heard
function Asr.heard(self: Asr, l: mut Listening, samples: Vec<f32>, last: bool): stringThe text heard in l after its next samples (last: the recording ends with them). The encoder is causal:
a frame reads mel frames up to 8 times its index and attends within its chunk of CHUNK frames and the chunks before,
so the frames of each whole chunk of settled mel frames are those of the whole recording, and their tokens are
final; the rest wait for more samples, or for the end.
Effects: fail, read, write, console, io, alloc.
Asr.encoded
function Asr.encoded(self: Asr, mel: Vec<f32>, prompt: i64): TensorThe frames of the encoder and the prompt projector for the decoder of the mel frames mel (rows of MELS): [T, 640].
Effects: fail, read, write, console, io, alloc.
Constants
MODEL
const MODEL = "nemotron-asr"The model's name in the catalog and as giz's worker serves it.
REPO
const REPO = "nvidia/nemotron-3.5-asr-streaming-0.6b"Its Hugging Face repository.