GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/written/memory

weights written one fact at a time, with no gradient and no training data kept. A Memory is a linear map from keys to values (an MLP's output weights: its keys are the MLP's activations). A write makes the map give a value for a key and keeps what it gives for every key it saw before: recursive least squares, which holds the inverse covariance of those keys. gizai/written/reader keeps its facts in one, read by cells of SHARP and THRESHOLD.

gizlang
import * as Memory from "gizai/written/memory";

Types

Memory

gizlang
type Memory

A map of keys inputs to values outputs, and the inverse covariance of the keys it saw.

  • keys: i64
  • values: i64
  • writes: i64
  • weights: Vec<f64>

Functions

memory

gizlang
function memory(weights: Tensor, ridge: f64): Memory

A memory of weights ([keys, values]) that saw no key yet: its inverse covariance is 1 / ridge. A larger ridge moves the weights less at a write and keeps more of a key it never saw.

Effects: fail, read, write, console, io, alloc.

decoded

gizlang
function decoded(data: bytes): Memory throws string

The memory that Memory.encode wrote.

Effects: fail, alloc.

Methods

Memory.read

gizlang
function Memory.read(self: Memory, key: Vec<f64>): Vec<f64>

What the map gives for key.

Effects: fail, alloc.

Memory.write

gizlang
function Memory.write(self: mut Memory, key: Vec<f64>, value: Vec<f64>): void

Makes the map give value for key and keeps what it gives for the keys before.

Effects: fail, alloc.

Memory.forget

gizlang
function Memory.forget(self: mut Memory, key: Vec<f64>, value: Vec<f64>): void

Takes back an earlier write of value for key: the map is as if that write never was, so a fact that changed is forgotten and written again, where a second write would give the mean of the two values. The key must be one that was written: taking back any other breaks the covariance.

Effects: fail, alloc.

Memory.set

gizlang
function Memory.set(self: mut Memory, key: Vec<f64>, value: Vec<f64>): void

Makes the map give value for key exactly, whether the key was written before or not: a fact that changed, with no need of its old value. The weights move along the key's reach by the whole error (a write over a held key moves them part of the way), and the key joins the covariance.

Effects: fail, alloc.

Memory.encode

gizlang
function Memory.encode(self: Memory): bytes

The memory as bytes: its sizes, then its weights and inverse covariance as f64, and decoded back.

Effects: fail, alloc.

Memory.tensor

gizlang
function Memory.tensor(self: Memory): Tensor

The weights as a tensor [keys, values].

Effects: fail, alloc.

Constants

SHARP

gizlang
const SHARP: f64 = 4.0

How sharply a memory cell turns on.

THRESHOLD

gizlang
const THRESHOLD: f64 = 1.5

The normed stream a memory cell turns on above.