# gizai:llm/sample

how a step's logits become tokens: Sampling, the shaping of the logits, grammars, and the check of drafts.

```gizlang
import * as Sample from "gizai:llm/sample";
```

## Types

### `Sampling`

```gizlang
type Sampling = { temperature: f64; topK: i64; topP: f64; seed: u64; presence?: f64; frequency?: f64; repetition?: f64; grammar?: Grammar; after?: i64; logprobs?: i64; }
```

How tokens are picked: temperature 0 the most likely, else one of the topK most likely (0: any) within topP of
their probability, at temperature. Before that the logits are shaped, as OpenAI and vLLM shape them: less by
`presence` for a token the answer has and by `frequency` for each time it has it; divided by `repetition` (above 0,
times it below 0) for a token the prompt or the answer has. `grammar`: only the tokens it allows (gizai:llm/grammar),
from the token after `after` on (a reasoning model's end of thinking; none: from the first); a stop token only where
the text is whole. `logprobs`: each token's log probability and the most likely ones (that many) are given, of the
model's logits before shaping. A shaped request makes no drafts.

- `temperature`: `f64`
- `topK`: `i64`
- `topP`: `f64`
- `seed`: `u64`
- `presence`: `_gizlang_sdk_core.Option<f64>`
- `frequency`: `_gizlang_sdk_core.Option<f64>`
- `repetition`: `_gizlang_sdk_core.Option<f64>`
- `grammar`: `_gizlang_sdk_core.Option<_gizlang_lib_ai__llm__grammar.Grammar>`
- `after`: `_gizlang_sdk_core.Option<i64>`
- `logprobs`: `_gizlang_sdk_core.Option<i64>`

### `Dist`

```gizlang
type Dist = { ids: Vec<i64>; weights: Vec<f64>; }
```

What a sampled token is drawn from: tokens, most likely first, and their weights (a token's probability is its
weight over their sum).

- `ids`: `Vec<i64>`
- `weights`: `Vec<f64>`

### `Request`

```gizlang
type Request = { prompt: Vec<i64>; limit: i64; sampling: Sampling; visual?: Visual; }
```

A prompt to continue: at most `limit` new tokens. With `visual` some of its rows are pictures' (Vision.chat): their
embeddings replace their tokens', and its rows take their three rotary positions from visual.grid (three for each of
its first rows, the last a text row's), the rows after them their position plus that last row's shift.

- `prompt`: `Vec<i64>`
- `limit`: `i64`
- `sampling`: `M53.Sampling`
- `visual`: `_gizlang_sdk_core.Option<M50.Visual>`

### `Logprob`

```gizlang
type Logprob = { token: i64; logprob: f64; top: Vec<i64>; tops: Vec<f64>; }
```

A token's log probability and the `top` most likely tokens with theirs, of the model's logits.

- `token`: `i64`
- `logprob`: `f64`
- `top`: `Vec<i64>`
- `tops`: `Vec<f64>`

## Functions

### `greedy`

```gizlang
function greedy(): Sampling
```

The most likely token at each step: the same tokens on each run.

### `best`

```gizlang
function best(logits: Vec<f32>, at: i64, vocab: i64): i64
```

The index of the largest of logits[at, at + vocab), the first of equal ones.

Effects: fail.

### `deviceDist`

```gizlang
function deviceDist(c: Vec<f32>, w: i64, s: Sampling, exact: bool): Dist
```

The distribution of wanted row w from a GPU's candidates of it (candidatesKernel). With `exact` it is empty where
the candidates may miss one of the row's top topK (a topK of 0 or above 256); without, a topK of 0 or above
DRAFT_TOPK takes the first DRAFT_TOPK (a draft may come from any distribution).

Effects: fail.

### `draw`

```gizlang
function draw(d: Dist, state: mut Vec<u64>): i64
```

A token of d, drawn with the random state.

Effects: fail.

### `chance`

```gizlang
function chance(d: Dist, token: i64): f64
```

The probability of `token` in d.

Effects: fail.

### `residual`

```gizlang
function residual(p: Dist, q: Dist): Dist
```

What follows a draft drawn from q that is not kept: max(0, p - q), so the tokens follow p (speculative sampling);
p where q covers all of it.

Effects: fail.

### `uniform`

```gizlang
function uniform(state: mut Vec<u64>): f64
```

A uniform number in [0, 1) from the random state.

Effects: fail.

### `pick`

```gizlang
function pick(logits: Vec<f32>, at: i64, vocab: i64, s: Sampling, state: mut Vec<u64>): i64
```

A token from logits[at, at + vocab); `state` holds the random state.

Effects: fail.

### `decide`

```gizlang
function decide(e: mut Engine, sys: System, tokens: Vec<i64>, slots: Vec<i64>, wanted: Vec<i64>, samplings: Vec<Sampling>, drafts: Vec<i64>): Decided throws string
```

Runs one step of `tokens` in `slots` and decides each `wanted` row: its top token, and its distribution or logits when its sequence samples or is shaped.

Effects: fail, read, write, console, io.

### `shaped`

```gizlang
function shaped(s: Sampling): bool
```

Whether a request's logits are shaped before a token is picked (Sampling).

### `shapedPick`

```gizlang
function shapedPick(e: mut Engine, row: Vec<f32>, s: Sampling, history: Vec<i64>, prompt: i64, reading: Option<Recognizer>, state: mut Vec<u64>): Logprob
```

The token a shaped request takes after `history` (its prompt's first `prompt` tokens, then its answer) from the
model's logits `row`, with its log probabilities (Sampling); `reading` is its grammar's recognizer while it holds.

Effects: fail.

gizai:llm/sample

how a step's logits become tokens: Sampling, the shaping of the logits, grammars, and the check of drafts.

gizlang

import * as Sample from "gizai:llm/sample";

Types

`Sampling`

gizlang

type Sampling = { temperature: f64; topK: i64; topP: f64; seed: u64; presence?: f64; frequency?: f64; repetition?: f64; grammar?: Grammar; after?: i64; logprobs?: i64; }

How tokens are picked: temperature 0 the most likely, else one of the topK most likely (0: any) within topP of their probability, at temperature. Before that the logits are shaped, as OpenAI and vLLM shape them: less by presence for a token the answer has and by frequency for each time it has it; divided by repetition (above 0, times it below 0) for a token the prompt or the answer has. grammar: only the tokens it allows (gizai:llm/grammar), from the token after after on (a reasoning model's end of thinking; none: from the first); a stop token only where the text is whole. logprobs: each token's log probability and the most likely ones (that many) are given, of the model's logits before shaping. A shaped request makes no drafts.

temperature: f64
topK: i64
topP: f64
seed: u64
presence: _gizlang_sdk_core.Option<f64>
frequency: _gizlang_sdk_core.Option<f64>
repetition: _gizlang_sdk_core.Option<f64>
grammar: _gizlang_sdk_core.Option<_gizlang_lib_ai__llm__grammar.Grammar>
after: _gizlang_sdk_core.Option<i64>
logprobs: _gizlang_sdk_core.Option<i64>

`Dist`

gizlang

type Dist = { ids: Vec<i64>; weights: Vec<f64>; }

What a sampled token is drawn from: tokens, most likely first, and their weights (a token's probability is its weight over their sum).

ids: Vec<i64>
weights: Vec<f64>

`Request`

gizlang

type Request = { prompt: Vec<i64>; limit: i64; sampling: Sampling; visual?: Visual; }

A prompt to continue: at most limit new tokens. With visual some of its rows are pictures' (Vision.chat): their embeddings replace their tokens', and its rows take their three rotary positions from visual.grid (three for each of its first rows, the last a text row's), the rows after them their position plus that last row's shift.

prompt: Vec<i64>
limit: i64
sampling: M53.Sampling
visual: _gizlang_sdk_core.Option<M50.Visual>

`Logprob`

gizlang

type Logprob = { token: i64; logprob: f64; top: Vec<i64>; tops: Vec<f64>; }

A token's log probability and the top most likely tokens with theirs, of the model's logits.

token: i64
logprob: f64
top: Vec<i64>
tops: Vec<f64>

Functions

`greedy`

gizlang

function greedy(): Sampling

The most likely token at each step: the same tokens on each run.

`best`

gizlang

function best(logits: Vec<f32>, at: i64, vocab: i64): i64

The index of the largest of logits[at, at + vocab), the first of equal ones.

Effects: fail.

`deviceDist`

gizlang

function deviceDist(c: Vec<f32>, w: i64, s: Sampling, exact: bool): Dist

The distribution of wanted row w from a GPU's candidates of it (candidatesKernel). With exact it is empty where the candidates may miss one of the row's top topK (a topK of 0 or above 256); without, a topK of 0 or above DRAFT_TOPK takes the first DRAFT_TOPK (a draft may come from any distribution).

Effects: fail.

`draw`

gizlang

function draw(d: Dist, state: mut Vec<u64>): i64

A token of d, drawn with the random state.

Effects: fail.

`chance`

gizlang

function chance(d: Dist, token: i64): f64

The probability of token in d.

Effects: fail.

`residual`

gizlang

function residual(p: Dist, q: Dist): Dist

What follows a draft drawn from q that is not kept: max(0, p - q), so the tokens follow p (speculative sampling); p where q covers all of it.

Effects: fail.

`uniform`

gizlang

function uniform(state: mut Vec<u64>): f64

A uniform number in [0, 1) from the random state.

Effects: fail.

`pick`

gizlang

function pick(logits: Vec<f32>, at: i64, vocab: i64, s: Sampling, state: mut Vec<u64>): i64

A token from logits[at, at + vocab); state holds the random state.

Effects: fail.

`decide`

gizlang

function decide(e: mut Engine, sys: System, tokens: Vec<i64>, slots: Vec<i64>, wanted: Vec<i64>, samplings: Vec<Sampling>, drafts: Vec<i64>): Decided throws string

Runs one step of tokens in slots and decides each wanted row: its top token, and its distribution or logits when its sequence samples or is shaped.

Effects: fail, read, write, console, io.

`shaped`

gizlang

function shaped(s: Sampling): bool

Whether a request's logits are shaped before a token is picked (Sampling).

`shapedPick`

gizlang

function shapedPick(e: mut Engine, row: Vec<f32>, s: Sampling, history: Vec<i64>, prompt: i64, reading: Option<Recognizer>, state: mut Vec<u64>): Logprob

The token a shaped request takes after history (its prompt's first prompt tokens, then its answer) from the model's logits row, with its log probabilities (Sampling); reading is its grammar's recognizer while it holds.

Effects: fail.