GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/written/facts

a GPT whose weights are written, never trained. It holds facts (subject, relation -> object) in the output weights of its MLPs, where one write stores one fact, and it follows a chain of relations hop by hop. The network is gizai/nn's GPT: World.config and Facts.params run in nn.gpt unchanged.

The residual stream holds named fields: the position (one-hot), the relation code and the entity code of the token, and the subject, the copy and the answer of the hop at the position. A field's values lie on rows of a Hadamard matrix that are orthogonal to the constant row, so a vector's mean is 0 and a layer norm only divides it by its length. Hop h works at position h + 2, the token of its relation, and nowhere else. Hop 0 is one layer: the attention copies the subject token's code into the subject field. A later hop is two layers. In the first the attention copies the answer of the hop before into the copy field, and the MLP writes each bit of the copy at its full value into the subject field, so the error of one hop does not reach the next. The second is the memory, as hop 0's MLP: a cell is on when two bits of the subject and two bits of the relation have its signs, so each (subject, relation) turns on its own set of cells, and the MLP's output weights map the set to the object's code. All hops hold one copy of the memory.

gizlang
import * as Facts from "gizai/written/facts";

Types

World

gizlang
type World = { entities: i64; relations: i64; hops: i64; code: i64; relationCode: i64; }

A network of entities and relations that follows chains of up to hops relations. An entity's code has code bits and a relation's relationCode bits.

  • entities: i64
  • relations: i64
  • hops: i64
  • code: i64
  • relationCode: i64

Query

gizlang
type Query = { subject: i64; relations: Vec<i64>; }

A question: the object at the end of relations from subject.

  • subject: i64
  • relations: Vec<i64>

Answer

gizlang
type Answer = { object: i64; score: f64; }

The entity the network names and how sure it is: near 1 for a fact it holds. For a fact it does not hold the score grows with the share of the cells in use, near its square root.

  • object: i64
  • score: f64

Facts

gizlang
type Facts

The network: its GPT parameters (the memory's place empty), the codes, the cells, the memory (cells to the bits of an object's code) and the object each (subject, relation) holds, -1 for none: what was written, which the memory's weights do not tell for sure once most cells are in use.

  • world: M50.World
  • memory: M51.Memory

Functions

facts

gizlang
function facts(k: Key, world: World): Facts

A network of world that holds no fact, its codes and cells drawn from k.

Effects: fail, read, write, console, io, alloc.

lastLogits

gizlang
function lastLogits(c: GptConfig, params: Vec<Tensor>, ids: Vec<i64>, rows: i64, t: i64): Tensor

The logits [rows, vocab] after the last token of each row of ids (rows rows of t tokens).

Effects: fail, read, write, console, io, alloc.

Methods

World.vocab

gizlang
function World.vocab(self: World): i64

The tokens: 0 starts a question, then the entities, then the relations.

Effects: fail.

World.config

gizlang
function World.config(self: World): GptConfig

The GPT of the world: one head, a layer for the first hop and two for each later one, and the smallest power of two that holds the fields.

Effects: fail.

World.cells

gizlang
function World.cells(self: World): i64

The cells of the memory: the MLP's width.

Effects: fail.

World.tokens

gizlang
function World.tokens(self: World, q: Query): Vec<i64>

The tokens of a question: the start, the subject and its relations. The token after them is the object.

Effects: fail, alloc.

World.entityToken

gizlang
function World.entityToken(self: World, entity: i64): i64

The token of an entity.

Effects: fail.

Facts.key

gizlang
function Facts.key(self: Facts, subject: i64, relation: i64): Vec<i64>

The cells that are on for the subject and the relation: the fact's key.

Effects: fail, alloc.

Facts.writes

gizlang
function Facts.writes(self: Facts): i64

The facts written and not replaced.

Facts.holds

gizlang
function Facts.holds(self: Facts, subject: i64, relation: i64): bool

Whether a fact of (subject, relation) was written.

Effects: fail.

Facts.learn

gizlang
function Facts.learn(self: mut Facts, subject: i64, relation: i64, object: i64): void

Writes the fact (subject, relation -> object) into the memory. A fact the pair holds is forgotten first, so the new one replaces it.

Effects: fail, alloc.

Facts.params

gizlang
function Facts.params(self: Facts): Vec<Tensor>

The GPT's parameters with the memory in every hop's MLP, in nn.gpt's order.

Effects: fail, read, write, console, io, alloc.

Facts.ask

gizlang
function Facts.ask(self: Facts, queries: Vec<Query>): Vec<Answer>

The answers of the network to queries, which all have as many relations.

Effects: fail, read, write, console, io, alloc.

Facts.study

gizlang
function Facts.study(self: mut Facts, queries: Vec<Query>, teacher: (subject: i64, relation: i64) => i64): i64

Learns what queries need: follows each one hop by hop, asks teacher for the object of every fact it does not hold, and writes it; a teacher that has no object gives -1, and the question ends there. The count of facts asked for.

Effects: fail, read, write, console, io, alloc.