GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/written/reader

a GPT that keeps its facts in written weights and is trained only to use them. The facts are the output weights of a memory of fixed random cells (./memory.giz). After each of its first layers the GPT asks the memory: a trained projection of the stream is the key, and the value comes back into the stream through another. A training step gives it a world it never saw, so its trained weights hold no fact: they learn to ask, to ask again with what came back (a chain of facts), and to say unknown when nothing was written. The GPT also writes: for a sentence it reads, the key is what it asks before the last word, and the value a projection at that word.

A fact is (entity, relation) to an entity. Each entity and relation has a fixed code of +1 and -1; a key is the sum of the two codes and a value the object's code. The codes and the cells are drawn from fixed keys, so a saved Reader is its trained parameters and its memory.

gizlang
import * as Reader from "gizai/written/reader";

Types

Form

gizlang
type Form = { config: GptConfig; reads: i64; code: i64; cells: i64; entities: i64; relations: i64; unknown: i64; ridge: f64; }

reads: the layers after which the GPT asks the memory (a chain of that many facts at most). code: the width of a key and a value. unknown: the word for a fact not written.

  • config: M22.GptConfig
  • reads: i64
  • code: i64
  • cells: i64
  • entities: i64
  • relations: i64
  • unknown: i64
  • ridge: f64

Ask

gizlang
type Ask = { prompt: Vec<i64>; answer: i64; asks: Vec<Vec<i64>>; }

A prompt, the word after it, and the fact (entity, relation) each read asks at the last token (none: empty).

  • prompt: Vec<i64>
  • answer: i64
  • asks: Vec<Vec<i64>>

Told

gizlang
type Told = { words: Vec<i64>; object: i64; }

A sentence up to the word of its object, and the object's entity. Its fact is what the first read asks at the word before.

  • words: Vec<i64>
  • object: i64

Reader

gizlang
type Reader

A trained reader of the small English and the memory it writes its facts into.

  • form: M52.Form
  • params: Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>
  • memory: M51.Memory

Keys

gizlang
type Keys = { facts: i64; across: Tensor; gram: Vec<f64>; }

The cells of each fact of a world, for Reader.written.

  • facts: i64
  • across: _gizlang_lib_ai__tensor__storage.Tensor<f32>
  • gram: Vec<f64>

Functions

reader

gizlang
function reader(k: Key, form: Form): Reader

A Reader with nothing trained and nothing written.

Effects: fail, read, write, console, io, alloc.

decoded

gizlang
function decoded(form: Form, data: bytes): Reader throws string

The reader of form that Reader.encode wrote.

Effects: fail, read, write, console, io, alloc.

Methods

Reader.onGpu

gizlang
function Reader.onGpu(self: mut Reader, gpu: Gpu): void

The Reader's tensors on gpu (training); a new Reader is on the host.

Effects: fail, read, write, console, io, alloc.

Reader.code

gizlang
function Reader.code(self: Reader, row: i64, i: i64): f64

Element i of the code of row: an entity, or a relation after the entities.

Effects: fail.

Reader.answers

gizlang
function Reader.answers(self: Reader, asks: Vec<Ask>): Vec<i64>

The word the Reader says after each prompt of asks (one length), with its memory as it is.

Effects: fail, read, write, console, io, alloc.

Reader.answersWith

gizlang
function Reader.answersWith(self: Reader, m: Tensor, asks: Vec<Ask>): Vec<i64>

answers with the memory m ([cells, code]) in place of its own.

Effects: fail, read, write, console, io, alloc.

Reader.read

gizlang
function Reader.read(self: mut Reader, told: Vec<Told>): void

The Reader reads each sentence of told (one length) and writes its fact into its memory, with no gradient. A fact written before has its new object from now on.

Effects: fail, read, write, console, io, alloc.

Reader.keys

gizlang
function Reader.keys(self: Reader, facts: Vec<Vec<i64>>): Keys

The cells of each of facts ((entity, relation) pairs), and their products with each other.

Effects: fail, read, write, console, io, alloc.

Reader.written

gizlang
function Reader.written(self: Reader, keys: Keys, known: Vec<bool>, objects: Vec<i64>): Tensor

The memory ([cells, code]) of a world: each fact of keys that known marks holds the code of its entity in objects, and the others were never written. It is what Memory gives for those writes one by one: the least squares solution through the facts' products (a Cholesky solve), in one step.

Effects: fail, read, write, console, io, alloc.

Reader.learn

gizlang
function Reader.learn(self: mut Reader, opt: mut nn.Adam, asks: Vec<Vec<Ask>>, told: Vec<Vec<Told>>, m: Tensor): Vec<f64>

One training step on a world whose memory is m: the answers of each set of asks (one length a set), the key of each read (taught: the fact's key), and the value of the last word of each set of told (taught: its entity's code). It gives the answers' loss, then the keys' and values'.

Effects: fail, read, write, console, io, alloc.

Reader.clear

gizlang
function Reader.clear(self: mut Reader): void

Forgets every written fact.

Effects: fail, read, write, console, io, alloc.

Reader.encode

gizlang
function Reader.encode(self: Reader): bytes

The Reader as bytes: its trained parameters (a run of gizai/nn) and its memory. decoded gives it back on the host.

Effects: fail, read, write, console, io, alloc.