# gizai:llm/encoders BERT-family encoders (BERT, RoBERTa, XLM-RoBERTa) for embeddings and ranking. ```gizlang import * as Encoders from "gizai:llm/encoders"; ``` ## Types ### `BertConfig` ```gizlang type BertConfig = { kind: string; layers: i64; heads: i64; dim: i64; eps: f64; act: string; positions: i64; offset: i64; } ``` A BERT-family encoder as a Hugging Face config.json gives it: BERT, RoBERTa and XLM-RoBERTa have one layout and differ in where positions start (`offset`: 0 for BERT, pad_token_id + 1 for RoBERTa). act is hidden_act: "gelu" (the erf form), "relu" or a tanh form ("gelu_new", "gelu_pytorch_tanh"). - `kind`: `string` - `layers`: `i64` - `heads`: `i64` - `dim`: `i64` - `eps`: `f64` - `act`: `string` - `positions`: `i64` - `offset`: `i64` ### `ModelFiles` ```gizlang type ModelFiles = (path: string) => Result, string> ``` Reads the file at a path of a model's repository: none when the model has no such file. ### `Encoder` ```gizlang type Encoder = { config: BertConfig; weights: Map; tokenizer: Tokenizer; pooling: string; normalize: bool; limit: i64; } ``` A sentence encoder as sentence-transformers 3 builds it from a model's files: an encoder, then the pooling of its token states ("mean", "cls", "max", "mean_sqrt_len" or "last"), then L2 normalization when the model has that module. - `config`: `M52.BertConfig` - `weights`: `_gizlang_sdk_collections.Map` - `tokenizer`: `M27.Tokenizer` - `pooling`: `string` - `normalize`: `bool` - `limit`: `i64` ### `CrossEncoder` ```gizlang type CrossEncoder = { config: BertConfig; weights: Map; tokenizer: Tokenizer; head: string; labels: i64; activation: string; limit: i64; } ``` A cross-encoder as sentence-transformers' CrossEncoder builds it: a sequence classification model that reads a pair of texts (a query and a document) and gives the scores of its labels. `head` is "bert" (BertForSequence- Classification: the pooler's tanh on the first token, then the classifier) or "roberta" (RobertaFor- and XLMRobertaForSequenceClassification: classifier.dense, tanh, classifier.out_proj). `activation` is the function on the scores: "sigmoid" (the default with one label), "identity" (the default with more) or "tanh". - `config`: `M52.BertConfig` - `weights`: `_gizlang_sdk_collections.Map` - `tokenizer`: `M27.Tokenizer` - `head`: `string` - `labels`: `i64` - `activation`: `string` - `limit`: `i64` ### `Ranked` ```gizlang type Ranked = { corpus: i64; score: f64; } ``` A document of a ranking: its index in the documents and its score. - `corpus`: `i64` - `score`: `f64` ## Functions ### `bertWeights` ```gizlang function bertWeights(tensors: Vec): Map ``` The tensors of a BertModel or RobertaModel checkpoint by their names, without the "bert." or "roberta." prefix of a task model, and with LayerNorm.gamma and beta (older checkpoints) as weight and bias. Effects: fail. ### `weight` ```gizlang function weight(w: Map, name: string): Tensor throws string ``` The tensor `name` of the weights `w`, or an error naming it. Effects: fail. ### `bert` ```gizlang function bert(c: BertConfig, w: Map, ids: Vec, types: Vec, sequences: i64): Tensor throws string ``` The last hidden states (sequences tokens, dim) of the encoder for `sequences` sequences of one length: their token ids and token types one sequence after another, every token attended (no padding). The weights may be on a GPU (Encoder.on); the states are where the weights are. Effects: fail, read, write, console, io. ### `heads` ```gizlang function heads(x: Tensor, n: i64, t: i64, h: i64): Tensor ``` The rows (n t, h hd) of n sequences as (n h, t, hd): each head of each sequence attends on its own. Effects: fail, read, write, console, io. ### `groups` ```gizlang function groups(encs: Vec, batch: i64): Vec> ``` Groups of the encodings with one token count, at most `batch` in a group, so a group goes through the encoder together with no padding: each group is the indexes of its encodings. Effects: fail. ### `encoderOf` ```gizlang function encoderOf(files: ModelFiles): Encoder throws string ``` The encoder of a model by the files it reads from its repository: config.json, tokenizer.json, model.safetensors and, for a sentence-transformers model, modules.json, sentence_bert_config.json and the pooling module's config.json. Without modules.json the tokens are mean-pooled and not normalized, as sentence-transformers does. Effects: fail, read, write, console, io. ### `crossEncoderOf` ```gizlang function crossEncoderOf(files: ModelFiles): CrossEncoder throws string ``` The cross-encoder of a model by its repository's files: config.json (its sentence_transformers.activation_fn, when present, names the activation), tokenizer.json and model.safetensors. Effects: fail, read, write, console, io. ## Methods ### `Encoder.on` ```gizlang function Encoder.on(self: Encoder, gpu: Gpu): Encoder ``` The encoder with its weights on `gpu`: it encodes there and gives the same embeddings to f32 precision. Effects: fail, read, write, console, io. ### `Encoder.embed` ```gizlang function Encoder.embed(self: Encoder, text: string, normalize: bool): Vec throws string ``` The embedding of a text (Encoder.embedBatch of one). Effects: fail, read, write, console, io. ### `Encoder.embedBatch` ```gizlang function Encoder.embedBatch(self: Encoder, texts: Vec, normalize: bool, batch: i64): Vec> throws string ``` The embeddings of texts: their token states pooled and, when the model says so or `normalize`, of length 1. The texts of one token count go through the encoder together, at most `batch` of them at a time, so no token is padding. Effects: fail, read, write, console, io. ### `CrossEncoder.on` ```gizlang function CrossEncoder.on(self: CrossEncoder, gpu: Gpu): CrossEncoder ``` The cross-encoder with its weights on `gpu`. Effects: fail, read, write, console, io. ### `CrossEncoder.scores` ```gizlang function CrossEncoder.scores(self: CrossEncoder, query: string, documents: Vec, batch: i64): Vec> throws string ``` The scores of the labels of each pair (query, documents[i]), after the activation: CrossEncoder.predict. The pairs of one token count go through the encoder together, at most `batch` at a time. Effects: fail, read, write, console, io. ### `CrossEncoder.rank` ```gizlang function CrossEncoder.rank(self: CrossEncoder, query: string, documents: Vec, top: i64, batch: i64): Vec throws string ``` The documents by the score of (query, document), the highest first, at most `top` of them (0: all): CrossEncoder. rank. Equal scores keep the documents' order. The model must have one label. Effects: fail, read, write, console, io.