GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/weights

the one reader of a model's stored weights. A checkpoint is safetensors (gizlang/safetensors gives each tensor's place in its file); this module puts a stored tensor where it runs: on the host as f32, on a GPU from the file straight to the device, a linear as BF16, ComfyUI's scaled FP8, raw codes, and ConvRot's INT8 and INT4 rows in the order the products read them. A model's loader names its tensors and calls these; it holds no reading of its own. A LoRA is gizai/lora's. The formats are gizai/quant's; an LLM's conversion is gizai/llm/convert.

gizlang
import * as Weights from "gizai/weights";

Functions

storedOn

gizlang
function storedOn(s: Stored, gpu: Option<Gpu>, linear: bool): Tensor throws string

A stored tensor where it runs: a linear as BF16 on a GPU, anything else as f32 (BF16, F16, F32 and E4M3 bytes go to the GPU as they are stored and are cast there).

Effects: fail, read, write, console, io, alloc.

storedHost

gizlang
function storedHost(s: Stored): Tensor throws string

A stored tensor on the host, as f32 values.

Effects: fail, read, write, console, io, alloc.

load

gizlang
function load(path: string, gpu: Gpu, kept: (name: string) => string = (name: string): string => name): Map<string, Tensor> throws string

The tensors of the file at path as f32 on gpu (storedHost), by the name kept gives each stored name (its name by default; "" leaves the tensor out): the one loader of a network that keeps every weight as f32.

Effects: fail, read, write, console, io, alloc.

hostAt

gizlang
function hostAt(named: Map<string, Stored>, name: string): Tensor throws string

Tensor name of a file's tensors by name, on the host (storedHost).

Effects: fail, read, write, console, io, alloc.

valuesAt

gizlang
function valuesAt(named: Map<string, Stored>, name: string): Vec<f32> throws string

The values of tensor name of a file's tensors by name.

Effects: fail, read, write, console, io, alloc.

weightAt

gizlang
function weightAt(named: Map<string, Stored>, name: string, gpu: Option<Gpu>, linear: bool, head: i64 = 0): Tensor throws string

Weight name where it runs (as storedOn). With head > 0 the weight is a fused QKV projection stored as H3's checkpoints hold it, a head's query, key and value rows together (heads, 3, head); its rows are reordered to all queries, all keys, all values first, the order of a LoRA's rows (gizai/lora) and of this module's attention.

Effects: fail, read, write, console, io, alloc.

convRotLinear

gizlang
function convRotLinear(named: Map<string, Stored>, base: string, from: Vec<i64>, gpu: Gpu, bits: i64 = 8): Linear throws string

The ConvRot linear base of a file (base.weight: I8 rows rotated by the regular Hadamard of 256, or 4-bit pairs a byte with 4 bits; base.weight_scale: a scale a row) as a Linear on gpu: its row r is the file's row from[r], its inputs in Linears.int8Rows' order (canonicalInto). Every ConvRot checkpoint's linear loads here.

Effects: fail, read, write, console, io, alloc.

rawAt

gizlang
function rawAt(s: Stored, gpu: Option<Gpu>, dtype: i64): Tensor throws string

A stored tensor of bytes (U8 or I8 codes, or E4M3 scales) where it runs: its bytes as dtype on a GPU, their values on the host (codes as unsigned bytes).

Effects: fail, read, write, console, io, alloc.

comfyInt8Rows

gizlang
function comfyInt8Rows(named: Map<string, Stored>, names: Vec<string>, gpu: Gpu): Vec<Tensor> throws string

Linears names of a ComfyUI INT8 ConvRot file (each name's .weight and .weight_scale: INT8 rows rotated by the regular Hadamard of 256, a scale a row) stacked by rows on gpu, as Linears.int8Rows' products read them (canonicalInto): [the INT8 weights [rows, in], their scales [1, rows] F32].

Effects: fail, read, write, console, io, alloc.

canonicalInto

gizlang
function canonicalInto(w: Tensor, from: Vec<i64>, out: Tensor, bits: i64 = 8): void

INT8 rows w ([out, in]) rotated in the regular Hadamard's own order, as ConvRot files store them, into out: its row r is w's row from[r], its inputs in the order Linears.int8Rows rotates inputs to (as W4.decodeInto's codes). With 4 bits w holds INT4 pairs ([out, in / 2] bytes, the even input in the low nibble: convrot_w4a4) and out is I4 [out, in].

Effects: fail, read, write, console, io, alloc.

Methods

Stored.upload

gizlang
function Stored.upload(self: Stored, gpu: Gpu): Buffer throws string

The stored tensor's raw bytes in a new buffer on gpu, read from its file straight to the device (Gpu.uploadFile).

Effects: fail, read, write, console, io, alloc.

Stored.writeTo

gizlang
function Stored.writeTo(self: Stored, gpu: Gpu, b: Buffer): void throws string

The stored tensor's raw bytes into the buffer b of gpu, from its file straight to the device (Gpu.readFiles).

Effects: fail, read, write, io, alloc.