# gizml:quant the formats of stored weights, apart from the model families and the kernels that run them. A format says how a linear's weights are stored: a codebook of 2 to 8 bits, F8, BF16 or F16, each block of GROUP inputs rotated (Rotation), in the order the matrix cores read it (Quantization); or W4, the checkpoint format of production's H3 (a 16-value codebook, E4M3 group scales, the regular Hadamard of netops' ROTATED); or K4 and K6, a GGUF file's Q4_K and Q6_K blocks as they are (K formats). This module encodes a format when a model is converted, and decodes it on the host and to the matrix cores' INT8. ```gizlang import * as Quant from "gizml:quant"; ``` ## Types ### `Packed` ```gizlang type Packed = { codes: bytes; scales: bytes; } ``` A row-block's codes and scales, as the file holds them. - `codes`: `bytes` - `scales`: `bytes` ### `KPacked` ```gizlang type KPacked = { codes: bytes; scales: bytes; subs: bytes; } ``` The rows of a block of 16 in K format f, as the file holds them. - `codes`: `bytes` - `scales`: `bytes` - `subs`: `bytes` ### `W4` ```gizlang type W4 = { groups: Tensor; codebook: Tensor; scale: Tensor; } ``` The decoding of a W4 linear, as production quantizes H3 (WanGP's asym_w4a8_int8 with ConvRot): its codes, two a byte (the low nibble first), index `codebook` (16 values); a code's value times the E4M3 scale of its 16 inputs (`groups`, [out, in / 16]), rounded to nearest even and held within ±127, is the INT8 weight, and `scale` ([1, out] f32) is each output's. Its inputs go through a Hadamard transform of 256 and an INT8 scale a row (gizml:netops int8Rows). - `groups`: `M32.Tensor` - `codebook`: `M32.Tensor` - `scale`: `M32.Tensor` ## Functions ### `signs` ```gizlang function signs(): Vec ``` The signs of D: fixed, the same for every block of every model. Effects: fail. ### `rotate` ```gizlang function rotate(v: mut Vec, at: i64, d: Vec): void ``` Rotates v[at, at + n) in place, n the length of d (a power of 2): H D v / sqrt(n). Effects: fail. ### `isFormat` ```gizlang function isFormat(f: i64): bool ``` Whether f is a format. ### `formatName` ```gizlang function formatName(f: i64): string ``` The name of format f: q2 to q8, bf16, f16, f8, k4 or k6. Effects: fail. ### `parseFormat` ```gizlang function parseFormat(name: string): Option ``` The format of `name`: formatName's (q2 to q8, bf16, f16, f8) or a number of codebook bits; none for another. Effects: fail. ### `codeBits` ```gizlang function codeBits(f: i64): i64 ``` The bits of a code of format f. ### `scaled` ```gizlang function scaled(f: i64): bool ``` Whether format f has a scale for each row of each block (a codebook or F8). ### `dtypeOf` ```gizlang function dtypeOf(f: i64): string ``` The safetensors dtype of format f's codes. ### `hostValue` ```gizlang function hostValue(f: i64, book: Vec, c: i64): f64 ``` The value of code c of format f, on the host: a codebook's from `book`. Effects: fail. ### `tileRow` ```gizlang function tileRow(g: i64, j: i64): i64 ``` The row (0..15) of value j of a lane in group g. Effects: fail. ### `tileCol` ```gizlang function tileCol(q: i64, j: i64): i64 ``` The input (0..15 within its tile) of value j of a lane at q. Effects: fail. ### `roundUp` ```gizlang function roundUp(n: i64, m: i64): i64 ``` N rounded up to a multiple of M. Effects: fail. ### `thresholdsOf` ```gizlang function thresholdsOf(book: Vec): Vec ``` The decision points between the sorted codebook's neighbors, padded to a multiple of 16 with a value above all. Effects: fail. ### `bookOf` ```gizlang function bookOf(sorted: Vec, bits: i64): Vec ``` A codebook of 2^bits values for the samples (sorted): Lloyd's algorithm in one dimension, where each value's samples are a run of the sorted list, started evenly spaced over the samples' range. (Started at the quantiles, the tails keep wide cells: at 8 bits the error of normal samples is 1.9 times the even start's.) Effects: fail. ### `packed` ```gizlang function packed(rows: Vec, at: i64, count: i64, k: i64, book: Vec, th: Vec, bits: i64, d: Vec): Packed ``` Quantizes 16 rows of k inputs (rows[at..], row after row; missing rows are zero) to `book`: each row's blocks rotated, then for each block of 128 a scale found by alternating the nearest codes and the least-squares scale. Effects: fail. ### `densePacked` ```gizlang function densePacked(rows: Vec, at: i64, count: i64, k: i64, f: i64, d: Vec): Packed ``` 16 rows of k inputs (as `packed`) in dense format f: each row's blocks rotated, then with F8 each block scaled so its largest value is 448. Effects: fail. ### `isK` ```gizlang function isK(f: i64): bool ``` Whether f is a K format. ### `kFormatOf` ```gizlang function kFormatOf(dtype: string): Option ``` The K format that stores GGUF block type `dtype` as it is (Q4_K, Q6_K), or none. ### `kSub` ```gizlang function kSub(f: i64): i64 ``` The inputs that share a sub-scale in K format f. ### `kPacked` ```gizlang function kPacked(data: bytes, count: i64, k: i64, f: i64): KPacked ``` 16 rows of k inputs from `data`, the GGUF blocks of `count` rows (row after row; missing rows are zero), in K format f. Effects: fail. ### `kValue` ```gizlang function kValue(f: i64, k: i64, scales: bytes, subs: Vec, rb: i64, row: i64, col: i64, q: i64): f64 ``` Weight `row` (0 to 15) of row block rb, input `col`, of a linear of k inputs in K format f, from its code q, the linear's scales and its subs as numbers, on the host. Effects: fail. ### `regularRotation` ```gizlang function regularRotation(gpu: Gpu): Tensor ``` The regular Hadamard rotation as a matrix ([256, 256] BF16 on `gpu`) of Kernel.hadamard: R[j][i] = regularSign(i) (-1)^popcount(i & j) / 16 (signs, then Sylvester's order). Effects: fail, read, write, console, io. ## Methods ### `W4.hostWeights` ```gizlang function W4.hostWeights(self: W4, w: Tensor): Tensor ``` The INT8 weights of W4 codes w ([out, in / 2], U8 values on the host) by this decoding, on the host, as F32 ([out, in]), each input o at its place after the regular rotation (K.regularSource, signed by K.regularSign): the order int8Rows rotates the inputs in, so the weights take x as int8Rows gives it. decodeInto on a GPU. Effects: fail. ### `W4.decodeInto` ```gizlang function W4.decodeInto(self: W4, w: Tensor, out: Tensor): void ``` The INT8 weights of W4 codes w ([out, in / 2]) by this decoding, into `out` ([out, in] I8). Effects: fail, read, write, console, io. ## Constants ### `GROUP` ```gizlang const GROUP = 128 ``` The inputs of a linear rotate and share a scale in blocks of this many. ### `ROTATED` ```gizlang const ROTATED = 256 ``` The values a regular Hadamard rotation of W4's inputs takes at once (ConvRot's group; netops int8Rows). ### `BF16` ```gizlang const BF16 = 16 ``` The format of dense bfloat16 values (see Formats); formats 2 to 8 are codebooks of that many bits. ### `F16` ```gizlang const F16 = 17 ``` The format of dense IEEE half-precision values. ### `F8` ```gizlang const F8 = 18 ``` The format of FP8 E4M3 values with block scales. ### `K4` ```gizlang const K4 = 20 ``` The format of GGUF's Q4_K blocks (see K formats). ### `K6` ```gizlang const K6 = 22 ``` The format of GGUF's Q6_K blocks (see K formats).