# gizml:tokenizer text into model inputs. The token ids that a Hugging Face tokenizer.json gives a text or a pair of texts, as the tokenizers 0.19 crate runs it. The added tokens are found first and are not normalized. Each other part goes through the normalizer, the pre-tokenizer splits it into words, and the model turns each word into vocabulary pieces: WordPiece takes the longest pieces (BERT), Unigram the most likely ones (SentencePiece models: XLM-RoBERTa, multilingual E5, BGE-M3, T5), BPE the merges in rank order (byte-level BPE of GPT-2, Llama 3, Qwen2, Mistral). The post-processor's template puts the special tokens around the pieces, and the decoder turns ids back into text. ```gizlang import * as Tokenizer from "gizml:tokenizer"; ``` ## Types ### `Tokenizer` ```gizlang type Tokenizer = { model: string; vocab: Map; unknown: i64; prefix: string; maxWord: i64; scores: Vec; longest: i64; lowest: f64; byteFallback: bool; ranks: Map; ignoreMerges: bool; normalizers: Vec; pre: Vec; decoders: Vec; pieces: Vec; added: Vec; addedIds: Vec; addedSpecial: Vec; single: Vec; pair: Vec; truncation: i64; } ``` A tokenizer.json, ready to encode. `model` is "WordPiece", "Unigram" or "BPE". WordPiece: the continuing prefix and the longest word. Unigram: the log probability of each piece by id, the most code points of a piece, the lowest probability, and whether an unknown piece becomes its bytes' tokens (also BPE's). BPE: the rank of each merge by its two pieces, and whether a word in the vocabulary is one token before any merge. `pieces`: the text of each id. - `model`: `string` - `vocab`: `_gizlang_sdk_collections.Map` - `unknown`: `i64` - `prefix`: `string` - `maxWord`: `i64` - `scores`: `Vec` - `longest`: `i64` - `lowest`: `f64` - `byteFallback`: `bool` - `ranks`: `_gizlang_sdk_collections.Map` - `ignoreMerges`: `bool` - `normalizers`: `Vec` - `pre`: `Vec` - `decoders`: `Vec` - `pieces`: `Vec` - `added`: `Vec` - `addedIds`: `Vec` - `addedSpecial`: `Vec` - `single`: `Vec` — The templates of one text and of a pair. - `pair`: `Vec` - `truncation`: `i64` — The longest encoding with its special tokens; 0: no limit. ### `Encoding` ```gizlang type Encoding = { ids: Vec; types: Vec; } ``` An encoded text: its ids and the token type of each. - `ids`: `Vec` - `types`: `Vec` ## Functions ### `parse` ```gizlang function parse(text: string): Tokenizer throws string ``` The tokenizer of a tokenizer.json text; an error names the part that is not read. Effects: fail. ## Methods ### `Tokenizer.encode` ```gizlang function Tokenizer.encode(self: Tokenizer, text: string, limit: i64): Encoding ``` The encoding of a text: the template's special tokens around its ids, which are cut to fit `limit` (0: the tokenizer's truncation) with them. Effects: fail. ### `Tokenizer.tokens` ```gizlang function Tokenizer.tokens(self: Tokenizer, text: string): Vec ``` The ids of a text alone, without the template's special tokens and uncut (a chat prompt, which has its own). Effects: fail. ### `Tokenizer.id` ```gizlang function Tokenizer.id(self: Tokenizer, token: string): Option ``` The id of a token's text, among the added tokens first, then the vocabulary. Effects: fail. ### `Tokenizer.decode` ```gizlang function Tokenizer.decode(self: Tokenizer, ids: Vec, skipSpecial: bool): string ``` The text of ids, as the tokenizer's decoder gives it; `skipSpecial`: without the special added tokens. Effects: fail. ### `Tokenizer.tokenBytes` ```gizlang function Tokenizer.tokenBytes(self: Tokenizer): Vec ``` The bytes each token adds to a decoded text in the middle of it, by id (Tokenizer.bytesOf). A sampler that constrains the text reads these (gizml:grammar). Effects: fail. ### `Tokenizer.bytesOf` ```gizlang function Tokenizer.bytesOf(self: Tokenizer, id: i64): bytes ``` The bytes token `id` adds to a decoded text in the middle of it (a special token: its text). The steps that only act at the text's start (Strip, Metaspace's first space) are left out, and bytes stay bytes: a token can hold part of a UTF-8 character. Effects: fail. ### `Tokenizer.encodePair` ```gizlang function Tokenizer.encodePair(self: Tokenizer, first: string, second: string, limit: i64): Encoding ``` The encoding of a pair of texts (a query and a document for a cross-encoder): the pair template around both, the longer one cut first to fit `limit` (0: the tokenizer's truncation), as truncation "longest_first" cuts. Effects: fail.