gizai/written/reader
a GPT that keeps its facts in written weights and is trained only to use them. The facts are the output weights of a memory of fixed random cells (./memory.giz). After each of its first layers the GPT asks the memory: a trained projection of the stream is the key, and the value comes back into the stream through another. A training step gives it a world it never saw, so its trained weights hold no fact: they learn to ask, to ask again with what came back (a chain of facts), and to say unknown when nothing was written. The GPT also writes: for a sentence it reads, the key is what it asks before the last word, and the value a projection at that word.
A fact is (entity, relation) to an entity. Each entity and relation has a fixed code of +1 and -1; a key is the sum of the two codes and a value the object's code. The codes and the cells are drawn from fixed keys, so a saved Reader is its trained parameters and its memory.
import * as Reader from "gizai/written/reader";Types
Form
type Form = { config: GptConfig; reads: i64; code: i64; cells: i64; entities: i64; relations: i64; unknown: i64; ridge: f64; }reads: the layers after which the GPT asks the memory (a chain of that many facts at most). code: the width of
a key and a value. unknown: the word for a fact not written.
config:M22.GptConfigreads:i64code:i64cells:i64entities:i64relations:i64unknown:i64ridge:f64
Ask
type Ask = { prompt: Vec<i64>; answer: i64; asks: Vec<Vec<i64>>; }A prompt, the word after it, and the fact (entity, relation) each read asks at the last token (none: empty).
prompt:Vec<i64>answer:i64asks:Vec<Vec<i64>>
Told
type Told = { words: Vec<i64>; object: i64; }A sentence up to the word of its object, and the object's entity. Its fact is what the first read asks at the word before.
words:Vec<i64>object:i64
Reader
type ReaderA trained reader of the small English and the memory it writes its facts into.
form:M52.Formparams:Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>memory:M51.Memory
Keys
type Keys = { facts: i64; across: Tensor; gram: Vec<f64>; }The cells of each fact of a world, for Reader.written.
facts:i64across:_gizlang_lib_ai__tensor__storage.Tensor<f32>gram:Vec<f64>
Functions
reader
function reader(k: Key, form: Form): ReaderA Reader with nothing trained and nothing written.
Effects: fail, read, write, console, io, alloc.
decoded
function decoded(form: Form, data: bytes): Reader throws stringThe reader of form that Reader.encode wrote.
Effects: fail, read, write, console, io, alloc.
Methods
Reader.onGpu
function Reader.onGpu(self: mut Reader, gpu: Gpu): voidThe Reader's tensors on gpu (training); a new Reader is on the host.
Effects: fail, read, write, console, io, alloc.
Reader.code
function Reader.code(self: Reader, row: i64, i: i64): f64Element i of the code of row: an entity, or a relation after the entities.
Effects: fail.
Reader.answers
function Reader.answers(self: Reader, asks: Vec<Ask>): Vec<i64>The word the Reader says after each prompt of asks (one length), with its memory as it is.
Effects: fail, read, write, console, io, alloc.
Reader.answersWith
function Reader.answersWith(self: Reader, m: Tensor, asks: Vec<Ask>): Vec<i64>answers with the memory m ([cells, code]) in place of its own.
Effects: fail, read, write, console, io, alloc.
Reader.read
function Reader.read(self: mut Reader, told: Vec<Told>): voidThe Reader reads each sentence of told (one length) and writes its fact into its memory, with no gradient. A fact
written before has its new object from now on.
Effects: fail, read, write, console, io, alloc.
Reader.keys
function Reader.keys(self: Reader, facts: Vec<Vec<i64>>): KeysThe cells of each of facts ((entity, relation) pairs), and their products with each other.
Effects: fail, read, write, console, io, alloc.
Reader.written
function Reader.written(self: Reader, keys: Keys, known: Vec<bool>, objects: Vec<i64>): TensorThe memory ([cells, code]) of a world: each fact of keys that known marks holds the code of its entity in
objects, and the others were never written. It is what Memory gives for those writes one by one: the least
squares solution through the facts' products (a Cholesky solve), in one step.
Effects: fail, read, write, console, io, alloc.
Reader.learn
function Reader.learn(self: mut Reader, opt: mut nn.Adam, asks: Vec<Vec<Ask>>, told: Vec<Vec<Told>>, m: Tensor): Vec<f64>One training step on a world whose memory is m: the answers of each set of asks (one length a set), the key
of each read (taught: the fact's key), and the value of the last word of each set of told (taught: its entity's
code). It gives the answers' loss, then the keys' and values'.
Effects: fail, read, write, console, io, alloc.
Reader.clear
function Reader.clear(self: mut Reader): voidForgets every written fact.
Effects: fail, read, write, console, io, alloc.
Reader.encode
function Reader.encode(self: Reader): bytesThe Reader as bytes: its trained parameters (a run of gizai/nn) and its memory. decoded gives it back on
the host.
Effects: fail, read, write, console, io, alloc.