GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/llm/score

how well a model reads text it did not train on. heldOut is the loss on held-out token shards (gizai/llm/learn Shards), in nats a token and bits a byte: the steadiest measure of a small model. choices reads files of minimal pairs (BLiMP's sentence_good and sentence_bad: the likelier sentence wins) or of questions (context, endings, label: the ending of the highest log probability a byte wins, as lm-eval's acc_norm), through the engine that serves the model.

gizlang
import * as Score from "gizai/llm/score";

Types

HeldOut

gizlang
type HeldOut = { step: i64; nats: f64; bits: f64; }

The held-out loss of a model: nats a token and bits a byte, at the step its folder holds.

  • step: i64
  • nats: f64
  • bits: f64

Choices

gizlang
type Choices = { file: string; right: i64; all: i64; }

The right answers of a file of choices, of all its lines.

  • file: string
  • right: i64
  • all: i64

Functions

heldOut

gizlang
function heldOut(folder: string, data: string, gpu: Option<Gpu>, context: i64 = 512): HeldOut throws string

The loss of the model in folder on the held-out shards of data: 512 batches of 32 contexts of context tokens (8.4 M tokens at 512), the same ones each time. The 131 k tokens of one training line differ by 2% from draw to draw, as much as a recipe's gain, so a comparison reads this.

Effects: fail, read, write, console, io, alloc.

choices

gizlang
function choices(folder: string, files: Vec<string>, host: bool = false): Vec<Choices> throws string

The model in folder (on the GPU, or on the host with host) on each file of choices. Each text starts after the token that parts documents, as a training context does.

Effects: fail, read, write, console, io, alloc.