gizai/chat
Chat request admission and output, used by local callers and HTTP serving.
import * as Chat from "gizai/chat";Types
Ask
type Ask = { limit: i64; stops: Vec<string>; effort: string; tools: Vec<JsonValue>; sampling?: Sampling; adapter?: string; }What a reply may do: its most tokens (0: the context's rest), the strings that end it, the reasoning effort ("": none; low, medium or high: a thinking model thinks first), the tools it may call (OpenAI's tool objects) and its sampling (none: the model's).
limit:i64stops:Vec<string>effort:stringtools:Vec<gizlang_json.Json>sampling:_gizlang_sdk_core.Option<M55.Sampling>adapter:_gizlang_sdk_core.Option<string>
Job
type Job = { at: i64; chat: ChatTokens; shown: Vec<Picture>; limit: i64; sampling: Sampling; stops: Vec<string>; thinking: bool; budget: i64; tools: Vec<JsonValue>; n: i64; index: i64; answer: Task.Channel<Piece>; adapter: string; }A chat request the server took: its prompt, its limit and sampling, the strings that end it, whether the model
thinks first and its most thinking tokens (OpenAi.effortBudget: at most half its limit, so the answer keeps room), the tools
it may call, its number, and the channel its answer's pieces go to; at its model (serve).
at:i64chat:M55.ChatTokensshown:Vec<M66.Picture>limit:i64sampling:M55.Samplingstops:Vec<string>thinking:boolbudget:i64tools:Vec<gizlang_json.Json>n:i64index:i64answer:gizlang_task.Channel<M66.Piece>adapter:string— The LoRA it runs with (Engine.adapt; "": none).
Piece
type Piece = { index: i64; text: string; reasoning: string; logprobs: Vec<Logprob>; calls: Vec<OpenAi.Call>; end: string; prompt: i64; cached: i64; tokens: i64; }A piece of an answer (choice index of a request): more of its text and reasoning and the log probabilities of
its tokens (when asked); the last also the tools it calls, why it ended ("stop", "tool_calls", "length", or "error"
with the message in text), and the prompt's tokens, those of them read from cache and the tokens written.
index:i64text:stringreasoning:stringlogprobs:Vec<M55.Logprob>calls:Vec<gizlang_openai.Call>end:stringprompt:i64cached:i64tokens:i64
Picture
type Picture = { shape: Vec<i64>; pixels: Vec<f32>; times: Vec<f64>; }A picture or a video's frames that a chat shows, as plain values (a tensor stays in the task that made it):
shape [3, h, w] or [n, 3, h, w], pixels in [0, 1], and a video's frame seconds (times).
shape:Vec<i64>pixels:Vec<f32>times:Vec<f64>
Seeing
type Seeing = { job: Job; begun: i64; done: Task.Reply<LlmVision.Seen, string>; }A pass of a job's pictures through the vision tower (seeing): the job, when it began, and where its result comes.
job:M66.Jobbegun:i64done:gizlang_task.Reply<_gizlang_lib_ai__llm__vision.Seen, string>
Live
type Live = { id: i64; job: Job; tokens: Vec<i64>; from: i64; done: i64; held: string; thinking: bool; started: bool; cached: i64; calling: string; shown: i64; begun: i64; pictures: i64; seeing: i64; first: i64; answered: i64; }A request the serving loop runs: its engine id and job, the tokens written, the first of them whose text was not sent and the one before it, the text held back (it may start a stop string, "</think>" or "<tool_call>"), whether the model still thinks, whether the answer started, the prompt's cached tokens (-1: not known yet), and the text from the first "<tool_call>" on (the calls, read at the end), and when its last piece was sent (ns); for its log line (logLine), when it was added, its pictures and the time to see them, and when its first piece and first answer text were sent (ns, 0: not yet).
id:i64job:M66.Jobtokens:Vec<i64>from:i64done:i64held:stringthinking:boolstarted:boolcached:i64calling:stringshown:i64begun:i64pictures:i64seeing:i64first:i64answered:i64
Functions
ready
function ready(m: Model, o: LlmEngine.EngineOptions = {}): Engine throws stringAn engine of m with options o (Model.engine), warm: each step shape compiled and tuned before the first request,
the tunings kept in the model's folder for the next start (Model.saveTunings).
Effects: fail, read, write, console, io, alloc.
ask
function ask(): AskAn Ask with no limit, no stops, no reasoning, no tools and the model's sampling.
Effects: fail, alloc.
reply
function reply(e: mut Engine, messages: Vec<Message>, a: Ask, each: (piece: Piece) => bool): Piece throws stringThe answer of e's model to messages, each piece given to each as it comes (false: stop there). Returns the
whole answer as one piece: its text, reasoning, tool calls, why it ended and the token counts.
Effects: fail, read, write, console, io, alloc.
noPiece
function noPiece(): PieceAn empty piece.
Effects: fail, alloc.
jobOf
function jobOf(m: Model, context: i64, messages: Vec<Message>, a: Ask, n: i64, buffered: i64, shown: Vec<Picture> = []): Job throws stringThe job of a chat of messages for model m with context positions (an error: the request is wrong); its answer
channel holds ANSWER_WINDOW pieces of each of buffered such jobs (0: one piece).
Effects: fail, read, write, io, alloc.
added
function added(e: mut Engine, job: Job): Live throws stringPuts job in turn on e (Engine.add) and returns its Live. A job that shows pictures waits for their pass
(seeing) here; a server starts the pass and steps its chats while it runs.
Effects: fail, read, write, console, io, alloc.
seeing
function seeing(e: mut Engine, job: Job, wake: Task.Channel<i64>): Seeing throws stringStarts the pass of what job shows through the model's vision tower (Vision.chat) in a task of its own, and
returns at once: the pass yields between its parts, so the chats step on while it runs. Its result comes on done,
and then wake gets a value. A fault of the pass (a failed GPU allocation is R004) fails this request only. One pass
at a time: it uses the room that pictureRoom frees.
Effects: fail, read, write, console, io, alloc.
seenWithin
function seenWithin(e: mut Engine, pass: Seeing, milliseconds: i64): Option<LlmVision.Seen> throws stringThe rows of a picture pass (Seeing) once they are ready within milliseconds, else none; a failed pass gives its
room on the card back and throws.
Effects: fail, read, write, console, io, alloc.
seenLive
function seenLive(e: mut Engine, pass: Seeing, seen: LlmVision.Seen): Live throws stringThe Live of a job whose pass ended with seen: its prompt with the pictures' rows, put in turn on e.
Effects: fail, read, write, console, io, alloc.
thinks
function thinks(m: Model, e: string): boolWhether model m thinks first at reasoning effort e.
promptJob
function promptJob(chat: ChatTokens, context: i64, a: Ask, thinking: bool, n: i64, buffered: i64): Job throws stringThe job of prompt chat (jobOf), the model thinking first when thinking.
Effects: fail, io, alloc.
served
function served(e: mut Engine, live0: Vec<Live>): Vec<Live>A step of an engine of the server: each live request's piece of it sent; returns the requests still live.
Effects: fail, read, write, console, io, alloc.
failedPiece
function failedPiece(message: string): PieceThe last piece of a failed answer: message as its text, its end "error".
Effects: fail, alloc.
failedStatus
function failedStatus(p: Piece): i64The HTTP status of a failed answer's last piece: 503 where the server had no room for the request now (another server, or this one later, answers it), else 500.
Effects: fail.
messageOf
function messageOf(m: Model, x: JsonValue, shown: mut Vec<Picture>): Message throws stringA message of a request: its role and text, an assistant's reasoning ("reasoning_content" or "reasoning") and its tool calls (their arguments as the text of a JSON object, or the object).
Effects: fail, read, write, console, io, alloc.