# gizai:llm/sample how a step's logits become tokens: Sampling, the shaping of the logits, grammars, and the check of drafts. ```gizlang import * as Sample from "gizai:llm/sample"; ``` ## Types ### `Sampling` ```gizlang type Sampling = { temperature: f64; topK: i64; topP: f64; seed: u64; presence?: f64; frequency?: f64; repetition?: f64; grammar?: Grammar; after?: i64; logprobs?: i64; } ``` How tokens are picked: temperature 0 the most likely, else one of the topK most likely (0: any) within topP of their probability, at temperature. Before that the logits are shaped, as OpenAI and vLLM shape them: less by `presence` for a token the answer has and by `frequency` for each time it has it; divided by `repetition` (above 0, times it below 0) for a token the prompt or the answer has. `grammar`: only the tokens it allows (gizai:llm/grammar), from the token after `after` on (a reasoning model's end of thinking; none: from the first); a stop token only where the text is whole. `logprobs`: each token's log probability and the most likely ones (that many) are given, of the model's logits before shaping. A shaped request makes no drafts. - `temperature`: `f64` - `topK`: `i64` - `topP`: `f64` - `seed`: `u64` - `presence`: `_gizlang_sdk_core.Option` - `frequency`: `_gizlang_sdk_core.Option` - `repetition`: `_gizlang_sdk_core.Option` - `grammar`: `_gizlang_sdk_core.Option<_gizlang_lib_ai__llm__grammar.Grammar>` - `after`: `_gizlang_sdk_core.Option` - `logprobs`: `_gizlang_sdk_core.Option` ### `Dist` ```gizlang type Dist = { ids: Vec; weights: Vec; } ``` What a sampled token is drawn from: tokens, most likely first, and their weights (a token's probability is its weight over their sum). - `ids`: `Vec` - `weights`: `Vec` ### `Request` ```gizlang type Request = { prompt: Vec; limit: i64; sampling: Sampling; visual?: Visual; } ``` A prompt to continue: at most `limit` new tokens. With `visual` some of its rows are pictures' (Vision.chat): their embeddings replace their tokens', and its rows take their three rotary positions from visual.grid (three for each of its first rows, the last a text row's), the rows after them their position plus that last row's shift. - `prompt`: `Vec` - `limit`: `i64` - `sampling`: `M53.Sampling` - `visual`: `_gizlang_sdk_core.Option` ### `Logprob` ```gizlang type Logprob = { token: i64; logprob: f64; top: Vec; tops: Vec; } ``` A token's log probability and the `top` most likely tokens with theirs, of the model's logits. - `token`: `i64` - `logprob`: `f64` - `top`: `Vec` - `tops`: `Vec` ## Functions ### `greedy` ```gizlang function greedy(): Sampling ``` The most likely token at each step: the same tokens on each run. ### `best` ```gizlang function best(logits: Vec, at: i64, vocab: i64): i64 ``` The index of the largest of logits[at, at + vocab), the first of equal ones. Effects: fail. ### `deviceDist` ```gizlang function deviceDist(c: Vec, w: i64, s: Sampling, exact: bool): Dist ``` The distribution of wanted row w from a GPU's candidates of it (candidatesKernel). With `exact` it is empty where the candidates may miss one of the row's top topK (a topK of 0 or above 256); without, a topK of 0 or above DRAFT_TOPK takes the first DRAFT_TOPK (a draft may come from any distribution). Effects: fail. ### `draw` ```gizlang function draw(d: Dist, state: mut Vec): i64 ``` A token of d, drawn with the random state. Effects: fail. ### `chance` ```gizlang function chance(d: Dist, token: i64): f64 ``` The probability of `token` in d. Effects: fail. ### `residual` ```gizlang function residual(p: Dist, q: Dist): Dist ``` What follows a draft drawn from q that is not kept: max(0, p - q), so the tokens follow p (speculative sampling); p where q covers all of it. Effects: fail. ### `uniform` ```gizlang function uniform(state: mut Vec): f64 ``` A uniform number in [0, 1) from the random state. Effects: fail. ### `pick` ```gizlang function pick(logits: Vec, at: i64, vocab: i64, s: Sampling, state: mut Vec): i64 ``` A token from logits[at, at + vocab); `state` holds the random state. Effects: fail. ### `decide` ```gizlang function decide(e: mut Engine, sys: System, tokens: Vec, slots: Vec, wanted: Vec, samplings: Vec, drafts: Vec): Decided throws string ``` Runs one step of `tokens` in `slots` and decides each `wanted` row: its top token, and its distribution or logits when its sequence samples or is shaped. Effects: fail, read, write, console, io. ### `shaped` ```gizlang function shaped(s: Sampling): bool ``` Whether a request's logits are shaped before a token is picked (Sampling). ### `shapedPick` ```gizlang function shapedPick(e: mut Engine, row: Vec, s: Sampling, history: Vec, prompt: i64, reading: Option, state: mut Vec): Logprob ``` The token a shaped request takes after `history` (its prompt's first `prompt` tokens, then its answer) from the model's logits `row`, with its log probabilities (Sampling); `reading` is its grammar's recognizer while it holds. Effects: fail.