# gizai/llm/score

how well a model reads text it did not train on. `heldOut` is the loss on held-out token shards (gizai/llm/learn Shards), in nats a token and bits a byte: the steadiest measure of a small model. `choices` reads files of minimal pairs (BLiMP's sentence_good and sentence_bad: the likelier sentence wins) or of questions (context, endings, label: the ending of the highest log probability a byte wins, as lm-eval's acc_norm), through the engine that serves the model.

```gizlang
import * as Score from "gizai/llm/score";
```

## Types

### `HeldOut`

```gizlang
type HeldOut = { step: i64; nats: f64; bits: f64; }
```

The held-out loss of a model: nats a token and bits a byte, at the step its folder holds.

- `step`: `i64`
- `nats`: `f64`
- `bits`: `f64`

### `Choices`

```gizlang
type Choices = { file: string; right: i64; all: i64; }
```

The right answers of a file of choices, of all its lines.

- `file`: `string`
- `right`: `i64`
- `all`: `i64`

## Functions

### `heldOut`

```gizlang
function heldOut(folder: string, data: string, gpu: Option<Gpu>, context: i64 = 512): HeldOut throws string
```

The loss of the model in `folder` on the held-out shards of `data`: 512 batches of 32 contexts of `context`
tokens (8.4 M tokens at 512), the same ones each time. The 131 k tokens of one training line differ by 2% from draw
to draw, as much as a recipe's gain, so a comparison reads this.

Effects: fail, read, write, console, io, alloc.

### `choices`

```gizlang
function choices(folder: string, files: Vec<string>, host: bool = false): Vec<Choices> throws string
```

The model in `folder` (on the GPU, or on the host with `host`) on each file of choices. Each text starts after
the token that parts documents, as a training context does.

Effects: fail, read, write, console, io, alloc.

