gizai/llm/score
how well a model reads text it did not train on. heldOut is the loss on held-out token shards (gizai/llm/learn Shards), in nats a token and bits a byte: the steadiest measure of a small model. choices reads files of minimal pairs (BLiMP's sentence_good and sentence_bad: the likelier sentence wins) or of questions (context, endings, label: the ending of the highest log probability a byte wins, as lm-eval's acc_norm), through the engine that serves the model.
import * as Score from "gizai/llm/score";Types
HeldOut
type HeldOut = { step: i64; nats: f64; bits: f64; }The held-out loss of a model: nats a token and bits a byte, at the step its folder holds.
step:i64nats:f64bits:f64
Choices
type Choices = { file: string; right: i64; all: i64; }The right answers of a file of choices, of all its lines.
file:stringright:i64all:i64
Functions
heldOut
function heldOut(folder: string, data: string, gpu: Option<Gpu>, context: i64 = 512): HeldOut throws stringThe loss of the model in folder on the held-out shards of data: 512 batches of 32 contexts of context
tokens (8.4 M tokens at 512), the same ones each time. The 131 k tokens of one training line differ by 2% from draw
to draw, as much as a recipe's gain, so a comparison reads this.
Effects: fail, read, write, console, io, alloc.
choices
function choices(folder: string, files: Vec<string>, host: bool = false): Vec<Choices> throws stringThe model in folder (on the GPU, or on the host with host) on each file of choices. Each text starts after
the token that parts documents, as a training context does.
Effects: fail, read, write, console, io, alloc.