gizai/written/facts
a GPT whose weights are written, never trained. It holds facts (subject, relation -> object) in the output weights of its MLPs, where one write stores one fact, and it follows a chain of relations hop by hop. The network is gizai/nn's GPT: World.config and Facts.params run in nn.gpt unchanged.
The residual stream holds named fields: the position (one-hot), the relation code and the entity code of the token, and the subject, the copy and the answer of the hop at the position. A field's values lie on rows of a Hadamard matrix that are orthogonal to the constant row, so a vector's mean is 0 and a layer norm only divides it by its length. Hop h works at position h + 2, the token of its relation, and nowhere else. Hop 0 is one layer: the attention copies the subject token's code into the subject field. A later hop is two layers. In the first the attention copies the answer of the hop before into the copy field, and the MLP writes each bit of the copy at its full value into the subject field, so the error of one hop does not reach the next. The second is the memory, as hop 0's MLP: a cell is on when two bits of the subject and two bits of the relation have its signs, so each (subject, relation) turns on its own set of cells, and the MLP's output weights map the set to the object's code. All hops hold one copy of the memory.
import * as Facts from "gizai/written/facts";Types
World
type World = { entities: i64; relations: i64; hops: i64; code: i64; relationCode: i64; }A network of entities and relations that follows chains of up to hops relations. An entity's code has
code bits and a relation's relationCode bits.
entities:i64relations:i64hops:i64code:i64relationCode:i64
Query
type Query = { subject: i64; relations: Vec<i64>; }A question: the object at the end of relations from subject.
subject:i64relations:Vec<i64>
Answer
type Answer = { object: i64; score: f64; }The entity the network names and how sure it is: near 1 for a fact it holds. For a fact it does not hold the score grows with the share of the cells in use, near its square root.
object:i64score:f64
Facts
type FactsThe network: its GPT parameters (the memory's place empty), the codes, the cells, the memory (cells to the bits of an object's code) and the object each (subject, relation) holds, -1 for none: what was written, which the memory's weights do not tell for sure once most cells are in use.
world:M50.Worldmemory:M51.Memory
Functions
facts
function facts(k: Key, world: World): FactsA network of world that holds no fact, its codes and cells drawn from k.
Effects: fail, read, write, console, io, alloc.
lastLogits
function lastLogits(c: GptConfig, params: Vec<Tensor>, ids: Vec<i64>, rows: i64, t: i64): TensorThe logits [rows, vocab] after the last token of each row of ids (rows rows of t tokens).
Effects: fail, read, write, console, io, alloc.
Methods
World.vocab
function World.vocab(self: World): i64The tokens: 0 starts a question, then the entities, then the relations.
Effects: fail.
World.config
function World.config(self: World): GptConfigThe GPT of the world: one head, a layer for the first hop and two for each later one, and the smallest power of two that holds the fields.
Effects: fail.
World.cells
function World.cells(self: World): i64The cells of the memory: the MLP's width.
Effects: fail.
World.tokens
function World.tokens(self: World, q: Query): Vec<i64>The tokens of a question: the start, the subject and its relations. The token after them is the object.
Effects: fail, alloc.
World.entityToken
function World.entityToken(self: World, entity: i64): i64The token of an entity.
Effects: fail.
Facts.key
function Facts.key(self: Facts, subject: i64, relation: i64): Vec<i64>The cells that are on for the subject and the relation: the fact's key.
Effects: fail, alloc.
Facts.writes
function Facts.writes(self: Facts): i64The facts written and not replaced.
Facts.holds
function Facts.holds(self: Facts, subject: i64, relation: i64): boolWhether a fact of (subject, relation) was written.
Effects: fail.
Facts.learn
function Facts.learn(self: mut Facts, subject: i64, relation: i64, object: i64): voidWrites the fact (subject, relation -> object) into the memory. A fact the pair holds is forgotten first, so the new one replaces it.
Effects: fail, alloc.
Facts.params
function Facts.params(self: Facts): Vec<Tensor>The GPT's parameters with the memory in every hop's MLP, in nn.gpt's order.
Effects: fail, read, write, console, io, alloc.
Facts.ask
function Facts.ask(self: Facts, queries: Vec<Query>): Vec<Answer>The answers of the network to queries, which all have as many relations.
Effects: fail, read, write, console, io, alloc.
Facts.study
function Facts.study(self: mut Facts, queries: Vec<Query>, teacher: (subject: i64, relation: i64) => i64): i64Learns what queries need: follows each one hop by hop, asks teacher for the object of every fact it does not
hold, and writes it; a teacher that has no object gives -1, and the question ends there. The count of facts
asked for.
Effects: fail, read, write, console, io, alloc.