gizai/written/memory
weights written one fact at a time, with no gradient and no training data kept. A Memory is a linear map from keys to values (an MLP's output weights: its keys are the MLP's activations). A write makes the map give a value for a key and keeps what it gives for every key it saw before: recursive least squares, which holds the inverse covariance of those keys. gizai/written/reader keeps its facts in one, read by cells of SHARP and THRESHOLD.
import * as Memory from "gizai/written/memory";Types
Memory
type MemoryA map of keys inputs to values outputs, and the inverse covariance of the keys it saw.
keys:i64values:i64writes:i64weights:Vec<f64>
Functions
memory
function memory(weights: Tensor, ridge: f64): MemoryA memory of weights ([keys, values]) that saw no key yet: its inverse covariance is 1 / ridge. A larger ridge
moves the weights less at a write and keeps more of a key it never saw.
Effects: fail, read, write, console, io, alloc.
decoded
function decoded(data: bytes): Memory throws stringThe memory that Memory.encode wrote.
Effects: fail, alloc.
Methods
Memory.read
function Memory.read(self: Memory, key: Vec<f64>): Vec<f64>What the map gives for key.
Effects: fail, alloc.
Memory.write
function Memory.write(self: mut Memory, key: Vec<f64>, value: Vec<f64>): voidMakes the map give value for key and keeps what it gives for the keys before.
Effects: fail, alloc.
Memory.forget
function Memory.forget(self: mut Memory, key: Vec<f64>, value: Vec<f64>): voidTakes back an earlier write of value for key: the map is as if that write never was, so a fact that changed is
forgotten and written again, where a second write would give the mean of the two values. The key must be one
that was written: taking back any other breaks the covariance.
Effects: fail, alloc.
Memory.set
function Memory.set(self: mut Memory, key: Vec<f64>, value: Vec<f64>): voidMakes the map give value for key exactly, whether the key was written before or not: a fact that changed, with
no need of its old value. The weights move along the key's reach by the whole error (a write over a held key
moves them part of the way), and the key joins the covariance.
Effects: fail, alloc.
Memory.encode
function Memory.encode(self: Memory): bytesThe memory as bytes: its sizes, then its weights and inverse covariance as f64, and decoded back.
Effects: fail, alloc.
Memory.tensor
function Memory.tensor(self: Memory): TensorThe weights as a tensor [keys, values].
Effects: fail, alloc.
Constants
SHARP
const SHARP: f64 = 4.0How sharply a memory cell turns on.
THRESHOLD
const THRESHOLD: f64 = 1.5The normed stream a memory cell turns on above.