gizml:tokenizer

text into model inputs. The token ids that a Hugging Face tokenizer.json gives a text or a pair of texts, as the tokenizers 0.19 crate runs it. The added tokens are found first and are not normalized. Each other part goes through the normalizer, the pre-tokenizer splits it into words, and the model turns each word into vocabulary pieces: WordPiece takes the longest pieces (BERT), Unigram the most likely ones (SentencePiece models: XLM-RoBERTa, multilingual E5, BGE-M3, T5), BPE the merges in rank order (byte-level BPE of GPT-2, Llama 3, Qwen2, Mistral). The post-processor's template puts the special tokens around the pieces, and the decoder turns ids back into text.

import * as Tokenizer from "gizml:tokenizer";

Types

Tokenizer

type Tokenizer = { model: string; vocab: Map<string, i64>; unknown: i64; prefix: string; maxWord: i64; scores: Vec<f64>; longest: i64; lowest: f64; byteFallback: bool; ranks: Map<string, i64>; ignoreMerges: bool; normalizers: Vec<Step>; pre: Vec<Step>; decoders: Vec<Step>; pieces: Vec<string>; added: Vec<string>; addedIds: Vec<i64>; addedSpecial: Vec<bool>; single: Vec<Piece>; pair: Vec<Piece>; truncation: i64; }

A tokenizer.json, ready to encode. model is "WordPiece", "Unigram" or "BPE". WordPiece: the continuing prefix and the longest word. Unigram: the log probability of each piece by id, the most code points of a piece, the lowest probability, and whether an unknown piece becomes its bytes' tokens (also BPE's). BPE: the rank of each merge by its two pieces, and whether a word in the vocabulary is one token before any merge. pieces: the text of each id.

• model: string
• vocab: _gizlang_sdk_collections.Map<string, i64>
• unknown: i64
• prefix: string
• maxWord: i64
• scores: Vec<f64>
• longest: i64
• lowest: f64
• byteFallback: bool
• ranks: _gizlang_sdk_collections.Map<string, i64>
• ignoreMerges: bool
• normalizers: Vec<M34.Step>
• pre: Vec<M34.Step>
• decoders: Vec<M34.Step>
• pieces: Vec<string>
• added: Vec<string>
• addedIds: Vec<i64>
• addedSpecial: Vec<bool>
• single: Vec<M34.Piece> — The templates of one text and of a pair.
• pair: Vec<M34.Piece>
• truncation: i64 — The longest encoding with its special tokens; 0: no limit.

Encoding

type Encoding = { ids: Vec<i64>; types: Vec<i64>; }

An encoded text: its ids and the token type of each.

• ids: Vec<i64>
• types: Vec<i64>

Functions

parse

function parse(text: string): Tokenizer throws string

The tokenizer of a tokenizer.json text; an error names the part that is not read.

Effects: fail.

Methods

Tokenizer.encode

function Tokenizer.encode(self: Tokenizer, text: string, limit: i64): Encoding

The encoding of a text: the template's special tokens around its ids, which are cut to fit limit (0: the tokenizer's truncation) with them.

Effects: fail.

Tokenizer.tokens

function Tokenizer.tokens(self: Tokenizer, text: string): Vec<i64>

The ids of a text alone, without the template's special tokens and uncut (a chat prompt, which has its own).

Effects: fail.

Tokenizer.id

function Tokenizer.id(self: Tokenizer, token: string): Option<i64>

The id of a token's text, among the added tokens first, then the vocabulary.

Effects: fail.

Tokenizer.decode

function Tokenizer.decode(self: Tokenizer, ids: Vec<i64>, skipSpecial: bool): string

The text of ids, as the tokenizer's decoder gives it; skipSpecial: without the special added tokens.

Effects: fail.

Tokenizer.tokenBytes

function Tokenizer.tokenBytes(self: Tokenizer): Vec<bytes>

The bytes each token adds to a decoded text in the middle of it, by id (Tokenizer.bytesOf). A sampler that constrains the text reads these (gizml:grammar).

Effects: fail.

Tokenizer.bytesOf

function Tokenizer.bytesOf(self: Tokenizer, id: i64): bytes

The bytes token id adds to a decoded text in the middle of it (a special token: its text). The steps that only act at the text's start (Strip, Metaspace's first space) are left out, and bytes stay bytes: a token can hold part of a UTF-8 character.

Effects: fail.

Tokenizer.encodePair

function Tokenizer.encodePair(self: Tokenizer, first: string, second: string, limit: i64): Encoding

The encoding of a pair of texts (a query and a document for a cross-encoder): the pair template around both, the longer one cut first to fit limit (0: the tokenizer's truncation), as truncation "longest_first" cuts.

Effects: fail.