GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/llm/corpus

the text a model trains on, from files to token shards. A file is Parquet (row groups read one at a time, so a file larger than memory packs), JSON lines with a "text" or a chat's "messages" each, or plain text whose documents an <|endoftext|> or an empty line parts. A chat is ChatML turns whose assistant turns a model learns; it reads the rest (the system, the user and tool turns) but takes no loss on them. tokenizer learns a byte-level BPE from the files, pack writes their tokens as the shards gizai/llm/learn trains on; every fiftieth document of a file is held out of both.

gizlang
import * as Corpus from "gizai/llm/corpus";

Types

Piece

gizlang
type Piece = { text: string; learned: bool; }

A piece of a document's text, which a model learns or only reads (a chat's prompt).

  • text: string
  • learned: bool

Functions

parts

gizlang
function parts(path: string): i64 throws string

The parts a file's documents are read in: a Parquet file's row groups, else one.

Effects: fail, io, alloc.

textOf

gizlang
function textOf(d: Vec<Piece>): string

A document's whole text.

Effects: fail, alloc.

documents

gizlang
function documents(path: string, part: i64): Vec<Vec<Piece>> throws string

The documents of part part of a file: a row group's texts or chats, .jsonl lines with a "text" or "messages" (role and content each) each, or a text whose documents <|endoftext|> or empty lines part. A row group is read alone, so a file larger than memory packs.

Effects: fail, read, io, alloc.

tokenizerOf

gizlang
function tokenizerOf(folder: string): Tok.Tokenizer throws string

The tokenizer that folder holds (its tokenizer.json).

Effects: fail, read, write, io, alloc.

held

gizlang
function held(n: i64): bool

Whether document n of a file, counted from 0, is held out: one in fifty, for the tokenizer and the tokens alike.

Effects: fail.

tokenizer

gizlang
function tokenizer(folder: string, size: i64, files: Vec<string>): string throws string

Learns the tokenizer of folder (size tokens, SPECIALS last) from at most 32 MB of the files' text, an equal share from each file, and says what it learned from.

Effects: fail, read, write, console, io, alloc.

pack

gizlang
function pack(folder: string, files: Vec<string>): Vec<string> throws string

Packs the files' documents, in the tokenizer of folder, as its shards train and held (gizai/llm/learn Shards): every fiftieth document of each file is held out, the ones tokenizer never reads. A piece that is only read is written quiet (its tokens plus the vocabulary). It says each shard set's tokens and bytes.

Effects: fail, read, write, console, io, alloc.