# gizai:omnivoice

k2-fsa's OmniVoice, text to speech in more than 600 languages, as its reference package and giz's worker run it: a voice from a short recording and its text (a clone), or from a few attributes (a design).

The model is Qwen3 0.6B as a bidirectional transformer over [text tokens | reference codes | target codes] of the Higgs codec (gizai:higgs: 8 codes of 1025 values, 1024 the mask, 25 frames a second). Decoding starts with every target code masked and unmasks a scheduled number of them each of 32 steps: the ones the model is most sure of, by classifier-free guidance against the target rows alone, less a penalty for the later codebooks, plus Gumbel noise. The blocks are gizai:diffusion's Block (its modulation still: no shift, no scale, gate 1), the key and value heads repeated to the query heads when they load.

```gizlang
import * as Omnivoice from "gizai:omnivoice";
```

## Types

### `OmniVoice`

```gizlang
type OmniVoice = { gpu: Gpu; arch: D.Arch; blocks: Vec<Block>; norm: Tensor; audio: Tensor; heads: Tensor; embed: Stored; tokenizer: Tokenizer; codec: H.Higgs; still: D.Modulation; zero: Tensor; inverse: Vec<f32>; }
```

The model on a GPU: its blocks, final norm, audio embeddings and heads (F32), the text embeddings where they are
stored (a request reads its rows), the tokenizer and the codec.

- `gpu`: `M10.Gpu`
- `arch`: `M22.Arch`
- `blocks`: `Vec<M22.Block>`
- `norm`: `M38.Tensor`
- `audio`: `M38.Tensor`
- `heads`: `M38.Tensor`
- `embed`: `gizlang_safetensors.Stored`
- `tokenizer`: `M40.Tokenizer`
- `codec`: `M26.Higgs`
- `still`: `M22.Modulation`
- `zero`: `M38.Tensor`
- `inverse`: `Vec<f32>`

### `Settings`

```gizlang
type Settings = { steps: i64; guidance: f64; seed: i64; denoise: bool; speed: f64; duration: f64; chunk: f64; threshold: f64; preprocess: bool; postprocess: bool; }
```

How a request is decoded and cut: the reference's OmniVoiceGenerationConfig.

- `steps`: `i64`
- `guidance`: `f64`
- `seed`: `i64`
- `denoise`: `bool`
- `speed`: `f64`
- `duration`: `f64`
- `chunk`: `f64`
- `threshold`: `f64`
- `preprocess`: `bool`
- `postprocess`: `bool`

### `Voice`

```gizlang
type Voice = { codes: Vec<Vec<i64>>; text: string; rms: f64; instruct: string; language: string; }
```

Who speaks: a clone's reference codes, text and RMS (none: a design, rms -1), or a design's instruction; and
the language id ("" none).

- `codes`: `Vec<Vec<i64>>`
- `text`: `string`
- `rms`: `f64`
- `instruct`: `string`
- `language`: `string`

### `Piece`

```gizlang
type Piece = { index: i64; total: i64; text: string; samples: Vec<f32>; }
```

A piece of speech: chunk `index` of `total`, its text, and its samples as decoded (24 kHz).

- `index`: `i64`
- `total`: `i64`
- `text`: `string`
- `samples`: `Vec<f32>`

### `Plan`

```gizlang
type Plan = { texts: Vec<string>; voice: Voice; instruct: string; target: i64; speed: f64; chunked: bool; first: Vec<Vec<i64>>; rng: Mt19937; settings: Settings; }
```

The chunks of a text in a voice and how long each is: the reference's chunking past s.threshold seconds. The
first chunk's codes are the voice of the next ones of a design.

- `texts`: `Vec<string>`
- `voice`: `M34.Voice`
- `instruct`: `string`
- `target`: `i64`
- `speed`: `f64`
- `chunked`: `bool`
- `first`: `Vec<Vec<i64>>`
- `rng`: `gizlang_random.Mt19937`
- `settings`: `M34.Settings`

### `Asked`

```gizlang
type Asked = { text: string; voice: Voice; settings: Settings; format: string; }
```

A request as giz's worker took it: the text, the voice, the settings and the stream format.

- `text`: `string`
- `voice`: `M34.Voice`
- `settings`: `M34.Settings`
- `format`: `string`

## Functions

### `load`

```gizlang
function load(sys: System, folder: string, gpu: Gpu): OmniVoice throws string
```

OmniVoice of the folder `folder` (k2-fsa/OmniVoice: model.safetensors, tokenizer.json and audio_tokenizer/) on
`gpu`.

Effects: fail, read, write, console, io.

### `settings`

```gizlang
function settings(): Settings
```

The reference's defaults for a file (chunks of 15 s past 30 s).

### `combinedText`

```gizlang
function combinedText(text: string, ref: string): string
```

The reference's _combine_text: the reference text and the text, without line breaks, Chinese parentheses as
ASCII, runs of spaces and tabs as one space, and no white space next to a CJK character.

Effects: fail.

### `estimatedFrames`

```gizlang
function estimatedFrames(text: string, ref: string, frames: i64, speed: f64): i64
```

The reference's estimate of the frames of `text` spoken as `ref` took `frames` ("Nice to meet you." in 25 without
a reference), boosted below 50 frames, over `speed`; at least 1.

Effects: fail.

### `chunks`

```gizlang
function chunks(text: string, most: i64, least: i64): Vec<string>
```

The reference's chunk_text_punctuation: sentences (cut after punctuation but abbreviations), merged up to
`most` characters, a chunk under `least` joined to the one before (the first to the second).

Effects: fail.

### `punctuated`

```gizlang
function punctuated(text: string): string
```

The reference's add_punctuation: a period ("。" in CJK text) after text that ends without one. The set is the
reference's as it parses.

Effects: fail.

### `languageOf`

```gizlang
function languageOf(language: string): string
```

The language id of a name or id ("Korean", "ko"), as the reference's _resolve_language; "" when none.

Effects: fail.

### `instruction`

```gizlang
function instruction(instruct: string, chinese: bool): string throws string
```

The instruction `instruct` checked and in one language (Chinese when `chinese` or a dialect asks, English when an
accent does), as the reference's _resolve_instruct; an error names an unknown or conflicting item.

Effects: fail.

### `withoutSilence`

```gizlang
function withoutSilence(x: Vec<f32>, middle: i64, lead: i64, trail: i64): Vec<f32>
```

The reference's remove_silence: silences longer than `middle` ms cut to it (split_on_silence), then the edges to
`lead` and `trail` ms.

Effects: fail.

### `fadedAndPadded`

```gizlang
function fadedAndPadded(x: Vec<f32>): Vec<f32>
```

The reference's fade_and_pad_audio: 0.1 s fades in and out (at most half each), then 0.1 s of silence on each side.

Effects: fail.

### `crossFaded`

```gizlang
function crossFaded(parts: Vec<Vec<f32>>): Vec<f32>
```

The reference's cross_fade_chunks: 0.1 s fades with 0.1 s of silence between chunks.

Effects: fail.

### `finished`

```gizlang
function finished(x: Vec<f32>, postprocess: bool, rms: f64): Vec<f32>
```

The reference's _post_process_audio: long silences out (with `postprocess`), the level of the reference's RMS
below 0.1 (`rms` < 0: no reference: a peak of 0.5), then fadedAndPadded.

Effects: fail.

### `designed`

```gizlang
function designed(instruct: string, language: string = ""): Voice
```

A designed voice of `instruct` ("female, young adult, moderate pitch": the worker's default).

Effects: fail.

### `plan`

```gizlang
function plan(text: string, voice: Voice, s: Settings): Plan throws string
```

The plan of speech of `text` in `voice`.

Effects: fail.

### `asked`

```gizlang
function asked(sys: System, m: OmniVoice, ffmpeg: string, dir: string, input: JsonValue, streaming: bool): Asked throws W.Failure
```

The request of a job's input or a realtime body, as giz's OmniVoice worker reads it (`streaming`: realtime).

Effects: fail, read, write, console, io.

### `serve`

```gizlang
function serve(sys: System, folder: string, id: string, host: string, port: i64, state: string, ffmpeg: string): i64 throws string
```

Serves OmniVoice of `folder` as giz's worker on host:port: audio.generate jobs (speech.wav) of model `id`
under `state`, and realtime speech on REALTIME (server-sent events), which goes before the jobs.

Effects: fail, read, write, console, io.

## Methods

### `OmniVoice.warm`

```gizlang
function OmniVoice.warm(self: OmniVoice, sys: System, folder: string, most: i64): void throws string
```

Tunes the products of every row count a request can have before the first one (one pass and one codec decode
for each of gizai:tensor's tuning steps up to `most` rows), the tunings kept in `folder`/tune for the next start.
Without it each new text length measured every product plan, 150 ms more for its request.

Effects: fail, read, write, console, io.

### `OmniVoice.cloned`

```gizlang
function OmniVoice.cloned(self: OmniVoice, samples: Vec<f32>, text: string, preprocess: bool = true, language: string = ""): Voice throws string
```

A cloned voice of the reference recording `samples` (24 kHz mono, its own level) that says `text`, as the
reference's create_voice_clone_prompt: raised to an RMS of 0.1, its silences cut (with `preprocess`), whole frames
coded.

Effects: fail, read, write, console, io.

### `OmniVoice.piece`

```gizlang
function OmniVoice.piece(self: OmniVoice, sys: System, p: mut Plan, i: i64, alive: () => bool): Piece throws string
```

Chunk i of plan p as decoded; `alive` is asked before each step (false: CANCELLED). Chunks go in order.

Effects: fail, read, write, console, io.

### `OmniVoice.spoken`

```gizlang
function OmniVoice.spoken(self: OmniVoice, sys: System, text: string, voice: Voice, s: Settings, alive: () => bool): Vec<f32> throws string
```

Speech of `text` in `voice` as one recording, the reference's file: pieces cross-faded, then finished.

Effects: fail, read, write, console, io.

## Constants

### `RATE`

```gizlang
const RATE = 24000
```

The samples a second of the speech.

### `CANCELLED`

```gizlang
const CANCELLED = "cancelled"
```

The error of a synthesis its caller stopped.

### `MODEL`

```gizlang
const MODEL = "omnivoice"
```

The model id giz gives OmniVoice.

### `REALTIME`

```gizlang
const REALTIME = "/v1/realtime/audio"
```

The route of realtime speech (server-sent events).