GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/written/edit

facts written into a language model's folder (one that gizai/llm/learn trains and saves), with no training. The model grows a last layer that starts as the identity: its attention's output and its MLP's down linear are 0. Each answer token to change gets one MLP cell, which turns on where the stream entering the layer is like that token's (a whitened likeness, the keys' covariance and bias taken out). The down linear is then solved on the GPU, by reweighted ridge least squares, to add at each such token the change of the stream that makes its next token lead the others by LEAD logits (found by gradient steps through the last norm and the head only), and to add nothing at every token of the chats to keep and of the model's own answers to them. The math emulates BF16, as the engine runs it.

gizlang
import * as Edit from "gizai/written/edit";

Types

Fact

gizlang
type Fact = { prompt: Vec<i64>; answer: Vec<i64>; text: string; }

A fact: a prompt in the chat template's turns, the answer's tokens (its closing turn too) and its text.

  • prompt: Vec<i64>
  • answer: Vec<i64>
  • text: string

Report

gizlang
type Report = { facts: i64; before: i64; after: i64; checks: i64; checkedBefore: i64; checkedAfter: i64; kept: i64; changed: i64; written: i64; held: i64; leading: i64; }

What an edit did: the facts answered right (the greedy answer starts with theirs) and the checks (the greedy answer holds their text) before and after, the answers to the kept chats that changed, and the tokens written and kept.

  • facts: i64
  • before: i64
  • after: i64
  • checks: i64
  • checkedBefore: i64
  • checkedAfter: i64
  • kept: i64
  • changed: i64
  • written: i64
  • held: i64
  • leading: i64

Functions

facts

gizlang
function facts(tok: Tok.Tokenizer, path: string): Vec<Fact> throws string

The facts of a JSON lines file: each line a prompt (or messages) and its answer, as giz ai train --tasks.

Effects: fail, read, write, io, alloc.

edit

gizlang
function edit(folder: string, out: string, written: Vec<Fact>, kept: Vec<Vec<i64>>, checks: Vec<Fact>, said: (line: string) => void, onGpu: bool = true): Report throws string

Writes facts into the model in folder and saves it to out, keeping the answers at every token of kept; checks (facts too) are only measured: questions the writes may answer, not written.

Effects: fail, read, write, console, io, alloc.

Constants

LEAD

gizlang
const LEAD: f64 = 4.0

The logits by which a fact's next token leads every other token, at each token of its answer, in the solve.