GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/tasks/dialogue

the spoken conversation task. A full-duplex model listens and speaks at once, a frame at a time: a family that has one gives a Talker (gizai/family's talker). giz ai talk holds a conversation aloud with it (this computer's microphone and speakers) or answers a recording, and a worker (dialogueService) serves it two ways: - a job (operation audio.dialogue): an uploaded recording of the person in, the model's side (reply.wav) out; - a conversation (TALK, a WebSocket): the first message is the opening as JSON (openingOf); then each binary message is the person's audio (16-bit little-endian mono PCM at the talker's rate, any length), and each frame the model says comes back at once as one binary message of the same form and, when it adds text, a text message {"type": "text", "text": ...}. The first message back is {"type": "started", "rate": R, "frame": F}.

gizlang
import * as Dialogue from "gizai/tasks/dialogue";

Types

Character

gizlang
type Character = { persona: string; name: string; gender: string; backchannel: string; other: string; relationship: string; situation: string; language: string; }

Who the model is in a conversation: its persona, name, gender ("female", "male" or ""), how often it backchannels ("low", "mid", "high" or ""), the other person's name, their relationship, the situation, and the language the card is written in ("en", "ko", "ja", "zh"; a model takes others as English). Empty fields are left out.

  • persona: string
  • name: string
  • gender: string
  • backchannel: string
  • other: string
  • relationship: string
  • situation: string
  • language: string

Opening

gizlang
type Opening = { character: Character; voice: Vec<f32>; speakFirst: bool; temperature: f64; guidance: f64; eagerness: f64; seed: u64; }

How a conversation opens: the character, a recording of the voice to speak with (mono samples at the talker's rate; none: the model's own), whether the model speaks first, the text's temperature, the audio's guidance and a bias toward starting to speak (each 0 for the model's own), and the seed (0: a random one).

  • character: M82.Character
  • voice: Vec<f32>
  • speakFirst: bool
  • temperature: f64
  • guidance: f64
  • eagerness: f64
  • seed: u64

Said

gizlang
type Said = { audio: Vec<f32>; text: string; }

A frame the model says: its audio (a frame of samples, -1 to 1) and the text it adds ("" for none).

  • audio: Vec<f32>
  • text: string

Talker

gizlang
type Talker = { gpu: Gpu; rate: i64; frame: i64; open: (o: Opening) => Said throws string; hear: (samples: Vec<f32>) => Said throws string; close: () => void; }

A full-duplex model on its card, one conversation at a time: its sample rate and frame, open starts a conversation and gives its first frame, hear takes the person's next frame and gives the model's next, and close ends it.

  • gpu: M7.Gpu
  • rate: i64
  • frame: i64
  • open: (o: M82.Opening) => M82.Said throws string
  • hear: (samples: Vec<f32>) => M82.Said throws string
  • close: () => void

Talked

gizlang
type Talked = { audio: Vec<f32>; text: string; }

The model's side of a conversation: its audio and the text it said.

  • audio: Vec<f32>
  • text: string

Functions

opening

gizlang
function opening(): Opening

An opening with no character and the model's own settings.

answer

gizlang
function answer(t: Talker, o: Opening, heard: Vec<f32>, progress: (done: i64, total: i64) => bool): Talked throws string

The model's side of a conversation with the person's recording heard (samples at its rate): a frame of the model for each frame heard and its first (the model is a frame ahead), and the text it said. progress hears the frames done; false from it stops.

Effects: fail, read, write, console, io, alloc.

openingOf

gizlang
function openingOf(j: JsonValue, ffmpeg: string, rate: i64): Opening throws string

The opening of a job's input or a conversation's first message: character (Character's fields; "other" may be "otherName"), voice (a recording: an uploaded file in a job, base64 bytes of any format in a conversation), speakFirst, temperature, guidance, eagerness, seed.

Effects: fail, read, write, console, io, alloc.

dialogueService

gizlang
function dialogueService(name: string, load: (gpu: Gpu) => Talker throws string, capacity: i64 = 4): Hosting.Service

The service of dialogue model name as giz's inference worker: jobs of operation audio.dialogue (input audio, an uploaded recording, and the opening's fields; output reply.wav, the model's side, and reply.txt, its words) and conversations on TALK. load makes the talker on the host's GPU.

Effects: fail, read, write, console, io, alloc.

Constants

TALK

gizlang
const TALK = "/v1/realtime/dialogue"

The streamed route of a conversation.