# gizai/llm/corpus

the text a model trains on, from files to token shards. A file is Parquet (row groups read one at a time, so a file larger than memory packs), JSON lines with a "text" or a chat's "messages" each, or plain text whose documents an <|endoftext|> or an empty line parts. A chat is ChatML turns whose assistant turns a model learns; it reads the rest (the system, the user and tool turns) but takes no loss on them. `tokenizer` learns a byte-level BPE from the files, `pack` writes their tokens as the shards gizai/llm/learn trains on; every fiftieth document of a file is held out of both.

```gizlang
import * as Corpus from "gizai/llm/corpus";
```

## Types

### `Piece`

```gizlang
type Piece = { text: string; learned: bool; }
```

A piece of a document's text, which a model learns or only reads (a chat's prompt).

- `text`: `string`
- `learned`: `bool`

## Functions

### `parts`

```gizlang
function parts(path: string): i64 throws string
```

The parts a file's documents are read in: a Parquet file's row groups, else one.

Effects: fail, io, alloc.

### `textOf`

```gizlang
function textOf(d: Vec<Piece>): string
```

A document's whole text.

Effects: fail, alloc.

### `documents`

```gizlang
function documents(path: string, part: i64): Vec<Vec<Piece>> throws string
```

The documents of part `part` of a file: a row group's texts or chats, `.jsonl` lines with a "text" or "messages"
(role and content each) each, or a text whose documents <|endoftext|> or empty lines part. A row group is read alone,
so a file larger than memory packs.

Effects: fail, read, io, alloc.

### `tokenizerOf`

```gizlang
function tokenizerOf(folder: string): Tok.Tokenizer throws string
```

The tokenizer that `folder` holds (its tokenizer.json).

Effects: fail, read, write, io, alloc.

### `held`

```gizlang
function held(n: i64): bool
```

Whether document `n` of a file, counted from 0, is held out: one in fifty, for the tokenizer and the tokens alike.

Effects: fail.

### `tokenizer`

```gizlang
function tokenizer(folder: string, size: i64, files: Vec<string>): string throws string
```

Learns the tokenizer of `folder` (`size` tokens, SPECIALS last) from at most 32 MB of the files' text, an equal
share from each file, and says what it learned from.

Effects: fail, read, write, console, io, alloc.

### `pack`

```gizlang
function pack(folder: string, files: Vec<string>): Vec<string> throws string
```

Packs the files' documents, in the tokenizer of `folder`, as its shards `train` and `held` (gizai/llm/learn
Shards): every fiftieth document of each file is held out, the ones `tokenizer` never reads. A piece that is only
read is written quiet (its tokens plus the vocabulary). It says each shard set's tokens and bytes.

Effects: fail, read, write, console, io, alloc.

