# gizai/models/nemotron/model

Nemotron 3.5 ASR Streaming 0.6B (nvidia/nemotron-3.5-asr-streaming-0.6b), speech to text in 40 languages, as transformers' Nemotron3_5AsrForRNNT runs a whole recording: the encoder's linears on the GPU when there is one, the rest on the CPU.

The log-mel features (gizai/models/nemotron/features) go through a causal subsampling of 8 (a 3x3 convolution at stride 2 to 256 channels, then two depthwise and pointwise stages, ReLU after each) and a linear to 1024 values a frame (80 ms). 24 FastConformer blocks follow: half a feed-forward, self-attention with relative positions (Transformer-XL's, with the content and position biases u and v) over chunks of 4 frames that see 14 chunks back, a convolution module (GLU, a causal depthwise convolution of 9 taps, layer norm, SiLU), half a feed-forward and a layer norm. A language prompt (a one-hot of 128) joins each frame through a two-layer projector, and a linear takes the frame to 640 values. The RNN-T decoder is an embedding and a 2-layer LSTM of 640; the joint adds a frame and the decoder's output, takes ReLU and a linear to the 13088 tokens (the last is blank). Greedy decoding emits up to 10 tokens a frame and moves to the next frame on blank.

```gizlang
import * as Model from "gizai/models/nemotron/model";
```

## Types

### `Asr`

```gizlang
type Asr = { named: Map<string, Tensor>; mel: Vec<f64>; tokenizer: Tokenizer; languages: Vec<string>; prompts: Vec<i64>; auto: i64; }
```

The model on the CPU: its tensors by name, the mel filters, the tokenizer and the language prompts.

- `named`: `_gizlang_sdk_collections.Map<string, _gizlang_lib_ai__tensor__storage.Tensor<f32>>`
- `mel`: `Vec<f64>`
- `tokenizer`: `M19.Tokenizer`
- `languages`: `Vec<string>`
- `prompts`: `Vec<i64>`
- `auto`: `i64`

### `Listening`

```gizlang
type Listening = { tail: Vec<f32>; at: i64; total: i64; mel: Vec<f32>; prompt: i64; decoder: Decoder; done: i64; ids: Vec<i64>; }
```

A recording heard as it comes: its samples from `at` on (those the unsettled mel frames read), how many came in
all, its settled mel frames, its language prompt, the decoder's state, the encoder frames decoded and their
tokens.

- `tail`: `Vec<f32>`
- `at`: `i64`
- `total`: `i64`
- `mel`: `Vec<f32>`
- `prompt`: `i64`
- `decoder`: `M83.Decoder`
- `done`: `i64`
- `ids`: `Vec<i64>`

## Functions

### `load`

```gizlang
function load(folder: string, gpu: Option<G.Gpu>): Asr throws string
```

The model of `folder` (model.safetensors, tokenizer.json, processor_config.json).

Effects: fail, read, write, console, io, alloc.

### `transcriber`

```gizlang
function transcriber(m: Asr): Transcriber
```

The speech-to-text engine of `m` (gizai/tasks/transcription).

Effects: fail, alloc.

### `service`

```gizlang
function service(folder: string, place: Place): Service
```

The service of the model of `folder` as giz's worker at `place` on gizai/host: audio.transcribe jobs, one at a
time, and speech heard as it is said on gizai/tasks/transcription's route, on the GPU where there is one.

Effects: fail, alloc.

## Methods

### `Asr.prompt`

```gizlang
function Asr.prompt(self: Asr, language: string): Option<i64>
```

The prompt of a language as the processor names it ("en-US", "de", "auto"), if the model has it.

Effects: fail.

### `Asr.transcribe`

```gizlang
function Asr.transcribe(self: Asr, samples: Vec<f32>, language: string): string throws string
```

The text spoken in `samples` (16 kHz, mono) in `language` ("" or "auto": the model finds it).

Effects: fail, read, write, console, io, alloc.

### `Asr.listening`

```gizlang
function Asr.listening(self: Asr, language: string): Listening throws string
```

A recording to hear as it comes, in `language` ("" or "auto": the model finds it).

Effects: fail, read, write, console, io, alloc.

### `Asr.heard`

```gizlang
function Asr.heard(self: Asr, l: mut Listening, samples: Vec<f32>, last: bool): string
```

The text heard in `l` after its next `samples` (`last`: the recording ends with them). The encoder is causal:
a frame reads mel frames up to 8 times its index and attends within its chunk of CHUNK frames and the chunks before,
so the frames of each whole chunk of settled mel frames are those of the whole recording, and their tokens are
final; the rest wait for more samples, or for the end.

Effects: fail, read, write, console, io, alloc.

### `Asr.encoded`

```gizlang
function Asr.encoded(self: Asr, mel: Vec<f32>, prompt: i64): Tensor
```

The frames of the encoder and the prompt projector for the decoder of the mel frames `mel` (rows of MELS): [T, 640].

Effects: fail, read, write, console, io, alloc.

## Constants

### `MODEL`

```gizlang
const MODEL = "nemotron-asr"
```

The model's name in the catalog and as giz's worker serves it.

### `REPO`

```gizlang
const REPO = "nvidia/nemotron-3.5-asr-streaming-0.6b"
```

Its Hugging Face repository.

