gizai/tasks/dialogue
the spoken conversation task. A full-duplex model listens and speaks at once, a frame at a time: a family that has one gives a Talker (gizai/family's talker). giz ai talk holds a conversation aloud with it (this computer's microphone and speakers) or answers a recording, and a worker (dialogueService) serves it two ways: - a job (operation audio.dialogue): an uploaded recording of the person in, the model's side (reply.wav) out; - a conversation (TALK, a WebSocket): the first message is the opening as JSON (openingOf); then each binary message is the person's audio (16-bit little-endian mono PCM at the talker's rate, any length), and each frame the model says comes back at once as one binary message of the same form and, when it adds text, a text message {"type": "text", "text": ...}. The first message back is {"type": "started", "rate": R, "frame": F}.
import * as Dialogue from "gizai/tasks/dialogue";Types
Character
type Character = { persona: string; name: string; gender: string; backchannel: string; other: string; relationship: string; situation: string; language: string; }Who the model is in a conversation: its persona, name, gender ("female", "male" or ""), how often it backchannels ("low", "mid", "high" or ""), the other person's name, their relationship, the situation, and the language the card is written in ("en", "ko", "ja", "zh"; a model takes others as English). Empty fields are left out.
persona:stringname:stringgender:stringbackchannel:stringother:stringrelationship:stringsituation:stringlanguage:string
Opening
type Opening = { character: Character; voice: Vec<f32>; speakFirst: bool; temperature: f64; guidance: f64; eagerness: f64; seed: u64; }How a conversation opens: the character, a recording of the voice to speak with (mono samples at the talker's rate; none: the model's own), whether the model speaks first, the text's temperature, the audio's guidance and a bias toward starting to speak (each 0 for the model's own), and the seed (0: a random one).
character:M82.Charactervoice:Vec<f32>speakFirst:booltemperature:f64guidance:f64eagerness:f64seed:u64
Said
type Said = { audio: Vec<f32>; text: string; }A frame the model says: its audio (a frame of samples, -1 to 1) and the text it adds ("" for none).
audio:Vec<f32>text:string
Talker
type Talker = { gpu: Gpu; rate: i64; frame: i64; open: (o: Opening) => Said throws string; hear: (samples: Vec<f32>) => Said throws string; close: () => void; }A full-duplex model on its card, one conversation at a time: its sample rate and frame, open starts a
conversation and gives its first frame, hear takes the person's next frame and gives the model's next, and
close ends it.
gpu:M7.Gpurate:i64frame:i64open:(o: M82.Opening) => M82.Said throws stringhear:(samples: Vec<f32>) => M82.Said throws stringclose:() => void
Talked
type Talked = { audio: Vec<f32>; text: string; }The model's side of a conversation: its audio and the text it said.
audio:Vec<f32>text:string
Functions
opening
function opening(): OpeningAn opening with no character and the model's own settings.
answer
function answer(t: Talker, o: Opening, heard: Vec<f32>, progress: (done: i64, total: i64) => bool): Talked throws stringThe model's side of a conversation with the person's recording heard (samples at its rate): a frame of the model
for each frame heard and its first (the model is a frame ahead), and the text it said. progress hears the frames
done; false from it stops.
Effects: fail, read, write, console, io, alloc.
openingOf
function openingOf(j: JsonValue, ffmpeg: string, rate: i64): Opening throws stringThe opening of a job's input or a conversation's first message: character (Character's fields; "other" may be
"otherName"), voice (a recording: an uploaded file in a job, base64 bytes of any format in a conversation),
speakFirst, temperature, guidance, eagerness, seed.
Effects: fail, read, write, console, io, alloc.
dialogueService
function dialogueService(name: string, load: (gpu: Gpu) => Talker throws string, capacity: i64 = 4): Hosting.ServiceThe service of dialogue model name as giz's inference worker: jobs of operation audio.dialogue (input audio,
an uploaded recording, and the opening's fields; output reply.wav, the model's side, and reply.txt, its words) and
conversations on TALK. load makes the talker on the host's GPU.
Effects: fail, read, write, console, io, alloc.
Constants
TALK
const TALK = "/v1/realtime/dialogue"The streamed route of a conversation.