GizStackGizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/chat

Chat request admission and output, used by local callers and HTTP serving.

gizlang
import * as Chat from "gizai/chat";

Types

Ask

gizlang
type Ask = { limit: i64; stops: Vec<string>; effort: string; tools: Vec<JsonValue>; sampling?: Sampling; adapter?: string; }

What a reply may do: its most tokens (0: the context's rest), the strings that end it, the reasoning effort ("": none; low, medium or high: a thinking model thinks first), the tools it may call (OpenAI's tool objects) and its sampling (none: the model's).

  • limit: i64
  • stops: Vec<string>
  • effort: string
  • tools: Vec<gizlang_json.Json>
  • sampling: _gizlang_sdk_core.Option<M55.Sampling>
  • adapter: _gizlang_sdk_core.Option<string>

Job

gizlang
type Job = { at: i64; chat: ChatTokens; shown: Vec<Picture>; limit: i64; sampling: Sampling; stops: Vec<string>; thinking: bool; budget: i64; tools: Vec<JsonValue>; n: i64; index: i64; answer: Task.Channel<Piece>; adapter: string; }

A chat request the server took: its prompt, its limit and sampling, the strings that end it, whether the model thinks first and its most thinking tokens (OpenAi.effortBudget: at most half its limit, so the answer keeps room), the tools it may call, its number, and the channel its answer's pieces go to; at its model (serve).

  • at: i64
  • chat: M55.ChatTokens
  • shown: Vec<M66.Picture>
  • limit: i64
  • sampling: M55.Sampling
  • stops: Vec<string>
  • thinking: bool
  • budget: i64
  • tools: Vec<gizlang_json.Json>
  • n: i64
  • index: i64
  • answer: gizlang_task.Channel<M66.Piece>
  • adapter: string — The LoRA it runs with (Engine.adapt; "": none).

Piece

gizlang
type Piece = { index: i64; text: string; reasoning: string; logprobs: Vec<Logprob>; calls: Vec<OpenAi.Call>; end: string; prompt: i64; cached: i64; tokens: i64; }

A piece of an answer (choice index of a request): more of its text and reasoning and the log probabilities of its tokens (when asked); the last also the tools it calls, why it ended ("stop", "tool_calls", "length", or "error" with the message in text), and the prompt's tokens, those of them read from cache and the tokens written.

  • index: i64
  • text: string
  • reasoning: string
  • logprobs: Vec<M55.Logprob>
  • calls: Vec<gizlang_openai.Call>
  • end: string
  • prompt: i64
  • cached: i64
  • tokens: i64

Picture

gizlang
type Picture = { shape: Vec<i64>; pixels: Vec<f32>; times: Vec<f64>; }

A picture or a video's frames that a chat shows, as plain values (a tensor stays in the task that made it): shape [3, h, w] or [n, 3, h, w], pixels in [0, 1], and a video's frame seconds (times).

  • shape: Vec<i64>
  • pixels: Vec<f32>
  • times: Vec<f64>

Seeing

gizlang
type Seeing = { job: Job; begun: i64; done: Task.Reply<LlmVision.Seen, string>; }

A pass of a job's pictures through the vision tower (seeing): the job, when it began, and where its result comes.

  • job: M66.Job
  • begun: i64
  • done: gizlang_task.Reply<_gizlang_lib_ai__llm__vision.Seen, string>

Live

gizlang
type Live = { id: i64; job: Job; tokens: Vec<i64>; from: i64; done: i64; held: string; thinking: bool; started: bool; cached: i64; calling: string; shown: i64; begun: i64; pictures: i64; seeing: i64; first: i64; answered: i64; }

A request the serving loop runs: its engine id and job, the tokens written, the first of them whose text was not sent and the one before it, the text held back (it may start a stop string, "</think>" or "<tool_call>"), whether the model still thinks, whether the answer started, the prompt's cached tokens (-1: not known yet), and the text from the first "<tool_call>" on (the calls, read at the end), and when its last piece was sent (ns); for its log line (logLine), when it was added, its pictures and the time to see them, and when its first piece and first answer text were sent (ns, 0: not yet).

  • id: i64
  • job: M66.Job
  • tokens: Vec<i64>
  • from: i64
  • done: i64
  • held: string
  • thinking: bool
  • started: bool
  • cached: i64
  • calling: string
  • shown: i64
  • begun: i64
  • pictures: i64
  • seeing: i64
  • first: i64
  • answered: i64

Functions

ready

gizlang
function ready(m: Model, o: LlmEngine.EngineOptions = {}): Engine throws string

An engine of m with options o (Model.engine), warm: each step shape compiled and tuned before the first request, the tunings kept in the model's folder for the next start (Model.saveTunings).

Effects: fail, read, write, console, io, alloc.

ask

gizlang
function ask(): Ask

An Ask with no limit, no stops, no reasoning, no tools and the model's sampling.

Effects: fail, alloc.

reply

gizlang
function reply(e: mut Engine, messages: Vec<Message>, a: Ask, each: (piece: Piece) => bool): Piece throws string

The answer of e's model to messages, each piece given to each as it comes (false: stop there). Returns the whole answer as one piece: its text, reasoning, tool calls, why it ended and the token counts.

Effects: fail, read, write, console, io, alloc.

noPiece

gizlang
function noPiece(): Piece

An empty piece.

Effects: fail, alloc.

jobOf

gizlang
function jobOf(m: Model, context: i64, messages: Vec<Message>, a: Ask, n: i64, buffered: i64, shown: Vec<Picture> = []): Job throws string

The job of a chat of messages for model m with context positions (an error: the request is wrong); its answer channel holds ANSWER_WINDOW pieces of each of buffered such jobs (0: one piece).

Effects: fail, read, write, io, alloc.

added

gizlang
function added(e: mut Engine, job: Job): Live throws string

Puts job in turn on e (Engine.add) and returns its Live. A job that shows pictures waits for their pass (seeing) here; a server starts the pass and steps its chats while it runs.

Effects: fail, read, write, console, io, alloc.

seeing

gizlang
function seeing(e: mut Engine, job: Job, wake: Task.Channel<i64>): Seeing throws string

Starts the pass of what job shows through the model's vision tower (Vision.chat) in a task of its own, and returns at once: the pass yields between its parts, so the chats step on while it runs. Its result comes on done, and then wake gets a value. A fault of the pass (a failed GPU allocation is R004) fails this request only. One pass at a time: it uses the room that pictureRoom frees.

Effects: fail, read, write, console, io, alloc.

seenWithin

gizlang
function seenWithin(e: mut Engine, pass: Seeing, milliseconds: i64): Option<LlmVision.Seen> throws string

The rows of a picture pass (Seeing) once they are ready within milliseconds, else none; a failed pass gives its room on the card back and throws.

Effects: fail, read, write, console, io, alloc.

seenLive

gizlang
function seenLive(e: mut Engine, pass: Seeing, seen: LlmVision.Seen): Live throws string

The Live of a job whose pass ended with seen: its prompt with the pictures' rows, put in turn on e.

Effects: fail, read, write, console, io, alloc.

thinks

gizlang
function thinks(m: Model, e: string): bool

Whether model m thinks first at reasoning effort e.

promptJob

gizlang
function promptJob(chat: ChatTokens, context: i64, a: Ask, thinking: bool, n: i64, buffered: i64): Job throws string

The job of prompt chat (jobOf), the model thinking first when thinking.

Effects: fail, io, alloc.

served

gizlang
function served(e: mut Engine, live0: Vec<Live>): Vec<Live>

A step of an engine of the server: each live request's piece of it sent; returns the requests still live.

Effects: fail, read, write, console, io, alloc.

failedPiece

gizlang
function failedPiece(message: string): Piece

The last piece of a failed answer: message as its text, its end "error".

Effects: fail, alloc.

failedStatus

gizlang
function failedStatus(p: Piece): i64

The HTTP status of a failed answer's last piece: 503 where the server had no room for the request now (another server, or this one later, answers it), else 500.

Effects: fail.

messageOf

gizlang
function messageOf(m: Model, x: JsonValue, shown: mut Vec<Picture>): Message throws string

A message of a request: its role and text, an assistant's reasoning ("reasoning_content" or "reasoning") and its tool calls (their arguments as the text of a JSON object, or the object).

Effects: fail, read, write, console, io, alloc.