gizai/llm/corpus
the text a model trains on, from files to token shards. A file is Parquet (row groups read one at a time, so a file larger than memory packs), JSON lines with a "text" or a chat's "messages" each, or plain text whose documents an <|endoftext|> or an empty line parts. A chat is ChatML turns whose assistant turns a model learns; it reads the rest (the system, the user and tool turns) but takes no loss on them. tokenizer learns a byte-level BPE from the files, pack writes their tokens as the shards gizai/llm/learn trains on; every fiftieth document of a file is held out of both.
import * as Corpus from "gizai/llm/corpus";Types
Piece
type Piece = { text: string; learned: bool; }A piece of a document's text, which a model learns or only reads (a chat's prompt).
text:stringlearned:bool
Functions
parts
function parts(path: string): i64 throws stringThe parts a file's documents are read in: a Parquet file's row groups, else one.
Effects: fail, io, alloc.
textOf
function textOf(d: Vec<Piece>): stringA document's whole text.
Effects: fail, alloc.
documents
function documents(path: string, part: i64): Vec<Vec<Piece>> throws stringThe documents of part part of a file: a row group's texts or chats, .jsonl lines with a "text" or "messages"
(role and content each) each, or a text whose documents <|endoftext|> or empty lines part. A row group is read alone,
so a file larger than memory packs.
Effects: fail, read, io, alloc.
tokenizerOf
function tokenizerOf(folder: string): Tok.Tokenizer throws stringThe tokenizer that folder holds (its tokenizer.json).
Effects: fail, read, write, io, alloc.
held
function held(n: i64): boolWhether document n of a file, counted from 0, is held out: one in fifty, for the tokenizer and the tokens alike.
Effects: fail.
tokenizer
function tokenizer(folder: string, size: i64, files: Vec<string>): string throws stringLearns the tokenizer of folder (size tokens, SPECIALS last) from at most 32 MB of the files' text, an equal
share from each file, and says what it learned from.
Effects: fail, read, write, console, io, alloc.
pack
function pack(folder: string, files: Vec<string>): Vec<string> throws stringPacks the files' documents, in the tokenizer of folder, as its shards train and held (gizai/llm/learn
Shards): every fiftieth document of each file is held out, the ones tokenizer never reads. A piece that is only
read is written quiet (its tokens plus the vocabulary). It says each shard set's tokens and bytes.
Effects: fail, read, write, console, io, alloc.