# gizml:quant

the formats of stored weights, apart from the model families and the kernels that run them. A format says how a linear's weights are stored: a codebook of 2 to 8 bits, F8, BF16 or F16, each block of GROUP inputs rotated (Rotation), in the order the matrix cores read it (Quantization); or W4, the checkpoint format of production's H3 (a 16-value codebook, E4M3 group scales, the regular Hadamard of netops' ROTATED); or K4 and K6, a GGUF file's Q4_K and Q6_K blocks as they are (K formats). This module encodes a format when a model is converted, and decodes it on the host and to the matrix cores' INT8.

```gizlang
import * as Quant from "gizml:quant";
```

## Types

### `Packed`

```gizlang
type Packed = { codes: bytes; scales: bytes; }
```

A row-block's codes and scales, as the file holds them.

- `codes`: `bytes`
- `scales`: `bytes`

### `KPacked`

```gizlang
type KPacked = { codes: bytes; scales: bytes; subs: bytes; }
```

The rows of a block of 16 in K format f, as the file holds them.

- `codes`: `bytes`
- `scales`: `bytes`
- `subs`: `bytes`

### `W4`

```gizlang
type W4 = { groups: Tensor; codebook: Tensor; scale: Tensor; }
```

The decoding of a W4 linear, as production quantizes H3 (WanGP's asym_w4a8_int8 with ConvRot): its codes, two a
byte (the low nibble first), index `codebook` (16 values); a code's value times the E4M3 scale of its 16 inputs
(`groups`, [out, in / 16]), rounded to nearest even and held within ±127, is the INT8 weight, and `scale` ([1, out]
f32) is each output's. Its inputs go through a Hadamard transform of 256 and an INT8 scale a row (gizml:netops int8Rows).

- `groups`: `M32.Tensor`
- `codebook`: `M32.Tensor`
- `scale`: `M32.Tensor`

## Functions

### `signs`

```gizlang
function signs(): Vec<f32>
```

The signs of D: fixed, the same for every block of every model.

Effects: fail.

### `rotate`

```gizlang
function rotate(v: mut Vec<f32>, at: i64, d: Vec<f32>): void
```

Rotates v[at, at + n) in place, n the length of d (a power of 2): H D v / sqrt(n).

Effects: fail.

### `isFormat`

```gizlang
function isFormat(f: i64): bool
```

Whether f is a format.

### `formatName`

```gizlang
function formatName(f: i64): string
```

The name of format f: q2 to q8, bf16, f16, f8, k4 or k6.

Effects: fail.

### `parseFormat`

```gizlang
function parseFormat(name: string): Option<i64>
```

The format of `name`: formatName's (q2 to q8, bf16, f16, f8) or a number of codebook bits; none for another.

Effects: fail.

### `codeBits`

```gizlang
function codeBits(f: i64): i64
```

The bits of a code of format f.

### `scaled`

```gizlang
function scaled(f: i64): bool
```

Whether format f has a scale for each row of each block (a codebook or F8).

### `dtypeOf`

```gizlang
function dtypeOf(f: i64): string
```

The safetensors dtype of format f's codes.

### `hostValue`

```gizlang
function hostValue(f: i64, book: Vec<f32>, c: i64): f64
```

The value of code c of format f, on the host: a codebook's from `book`.

Effects: fail.

### `tileRow`

```gizlang
function tileRow(g: i64, j: i64): i64
```

The row (0..15) of value j of a lane in group g.

Effects: fail.

### `tileCol`

```gizlang
function tileCol(q: i64, j: i64): i64
```

The input (0..15 within its tile) of value j of a lane at q.

Effects: fail.

### `roundUp`

```gizlang
function roundUp(n: i64, m: i64): i64
```

N rounded up to a multiple of M.

Effects: fail.

### `thresholdsOf`

```gizlang
function thresholdsOf(book: Vec<f64>): Vec<f32>
```

The decision points between the sorted codebook's neighbors, padded to a multiple of 16 with a value above all.

Effects: fail.

### `bookOf`

```gizlang
function bookOf(sorted: Vec<f64>, bits: i64): Vec<f64>
```

A codebook of 2^bits values for the samples (sorted): Lloyd's algorithm in one dimension, where each value's
samples are a run of the sorted list, started evenly spaced over the samples' range. (Started at the quantiles,
the tails keep wide cells: at 8 bits the error of normal samples is 1.9 times the even start's.)

Effects: fail.

### `packed`

```gizlang
function packed(rows: Vec<f32>, at: i64, count: i64, k: i64, book: Vec<f64>, th: Vec<f32>, bits: i64, d: Vec<f32>): Packed
```

Quantizes 16 rows of k inputs (rows[at..], row after row; missing rows are zero) to `book`: each row's blocks
rotated, then for each block of 128 a scale found by alternating the nearest codes and the least-squares scale.

Effects: fail.

### `densePacked`

```gizlang
function densePacked(rows: Vec<f32>, at: i64, count: i64, k: i64, f: i64, d: Vec<f32>): Packed
```

16 rows of k inputs (as `packed`) in dense format f: each row's blocks rotated, then with F8 each block scaled so its
largest value is 448.

Effects: fail.

### `isK`

```gizlang
function isK(f: i64): bool
```

Whether f is a K format.

### `kFormatOf`

```gizlang
function kFormatOf(dtype: string): Option<i64>
```

The K format that stores GGUF block type `dtype` as it is (Q4_K, Q6_K), or none.

### `kSub`

```gizlang
function kSub(f: i64): i64
```

The inputs that share a sub-scale in K format f.

### `kPacked`

```gizlang
function kPacked(data: bytes, count: i64, k: i64, f: i64): KPacked
```

16 rows of k inputs from `data`, the GGUF blocks of `count` rows (row after row; missing rows are zero), in K format
f.

Effects: fail.

### `kValue`

```gizlang
function kValue(f: i64, k: i64, scales: bytes, subs: Vec<f32>, rb: i64, row: i64, col: i64, q: i64): f64
```

Weight `row` (0 to 15) of row block rb, input `col`, of a linear of k inputs in K format f, from its code q, the
linear's scales and its subs as numbers, on the host.

Effects: fail.

### `regularRotation`

```gizlang
function regularRotation(gpu: Gpu): Tensor
```

The regular Hadamard rotation as a matrix ([256, 256] BF16 on `gpu`) of Kernel.hadamard: R[j][i] =
regularSign(i) (-1)^popcount(i & j) / 16 (signs, then Sylvester's order).

Effects: fail, read, write, console, io.

## Methods

### `W4.hostWeights`

```gizlang
function W4.hostWeights(self: W4, w: Tensor): Tensor
```

The INT8 weights of W4 codes w ([out, in / 2], U8 values on the host) by this decoding, on the host, as F32
([out, in]), each input o at its place after the regular rotation (K.regularSource, signed by K.regularSign): the
order int8Rows rotates the inputs in, so the weights take x as int8Rows gives it. decodeInto on a GPU.

Effects: fail.

### `W4.decodeInto`

```gizlang
function W4.decodeInto(self: W4, w: Tensor, out: Tensor): void
```

The INT8 weights of W4 codes w ([out, in / 2]) by this decoding, into `out` ([out, in] I8).

Effects: fail, read, write, console, io.

## Constants

### `GROUP`

```gizlang
const GROUP = 128
```

The inputs of a linear rotate and share a scale in blocks of this many.

### `ROTATED`

```gizlang
const ROTATED = 256
```

The values a regular Hadamard rotation of W4's inputs takes at once (ConvRot's group; netops int8Rows).

### `BF16`

```gizlang
const BF16 = 16
```

The format of dense bfloat16 values (see Formats); formats 2 to 8 are codebooks of that many bits.

### `F16`

```gizlang
const F16 = 17
```

The format of dense IEEE half-precision values.

### `F8`

```gizlang
const F8 = 18
```

The format of FP8 E4M3 values with block scales.

### `K4`

```gizlang
const K4 = 20
```

The format of GGUF's Q4_K blocks (see K formats).

### `K6`

```gizlang
const K6 = 22
```

The format of GGUF's Q6_K blocks (see K formats).

gizml:quant

the formats of stored weights, apart from the model families and the kernels that run them. A format says how a linear's weights are stored: a codebook of 2 to 8 bits, F8, BF16 or F16, each block of GROUP inputs rotated (Rotation), in the order the matrix cores read it (Quantization); or W4, the checkpoint format of production's H3 (a 16-value codebook, E4M3 group scales, the regular Hadamard of netops' ROTATED); or K4 and K6, a GGUF file's Q4_K and Q6_K blocks as they are (K formats). This module encodes a format when a model is converted, and decodes it on the host and to the matrix cores' INT8.

gizlang

import * as Quant from "gizml:quant";

Types

`Packed`

gizlang

type Packed = { codes: bytes; scales: bytes; }

A row-block's codes and scales, as the file holds them.

codes: bytes
scales: bytes

`KPacked`

gizlang

type KPacked = { codes: bytes; scales: bytes; subs: bytes; }

The rows of a block of 16 in K format f, as the file holds them.

codes: bytes
scales: bytes
subs: bytes

`W4`

gizlang

type W4 = { groups: Tensor; codebook: Tensor; scale: Tensor; }

The decoding of a W4 linear, as production quantizes H3 (WanGP's asym_w4a8_int8 with ConvRot): its codes, two a byte (the low nibble first), index codebook (16 values); a code's value times the E4M3 scale of its 16 inputs (groups, [out, in / 16]), rounded to nearest even and held within ±127, is the INT8 weight, and scale ([1, out] f32) is each output's. Its inputs go through a Hadamard transform of 256 and an INT8 scale a row (gizml:netops int8Rows).

groups: M32.Tensor
codebook: M32.Tensor
scale: M32.Tensor

Functions

`signs`

gizlang

function signs(): Vec<f32>

The signs of D: fixed, the same for every block of every model.

Effects: fail.

`rotate`

gizlang

function rotate(v: mut Vec<f32>, at: i64, d: Vec<f32>): void

Rotates v[at, at + n) in place, n the length of d (a power of 2): H D v / sqrt(n).

Effects: fail.

`isFormat`

gizlang

function isFormat(f: i64): bool

Whether f is a format.

`formatName`

gizlang

function formatName(f: i64): string

The name of format f: q2 to q8, bf16, f16, f8, k4 or k6.

Effects: fail.

`parseFormat`

gizlang

function parseFormat(name: string): Option<i64>

The format of name: formatName's (q2 to q8, bf16, f16, f8) or a number of codebook bits; none for another.

Effects: fail.

`codeBits`

gizlang

function codeBits(f: i64): i64

The bits of a code of format f.

`scaled`

gizlang

function scaled(f: i64): bool

Whether format f has a scale for each row of each block (a codebook or F8).

`dtypeOf`

gizlang

function dtypeOf(f: i64): string

The safetensors dtype of format f's codes.

`hostValue`

gizlang

function hostValue(f: i64, book: Vec<f32>, c: i64): f64

The value of code c of format f, on the host: a codebook's from book.

Effects: fail.

`tileRow`

gizlang

function tileRow(g: i64, j: i64): i64

The row (0..15) of value j of a lane in group g.

Effects: fail.

`tileCol`

gizlang

function tileCol(q: i64, j: i64): i64

The input (0..15 within its tile) of value j of a lane at q.

Effects: fail.

`roundUp`

gizlang

function roundUp(n: i64, m: i64): i64

N rounded up to a multiple of M.

Effects: fail.

`thresholdsOf`

gizlang

function thresholdsOf(book: Vec<f64>): Vec<f32>

The decision points between the sorted codebook's neighbors, padded to a multiple of 16 with a value above all.

Effects: fail.

`bookOf`

gizlang

function bookOf(sorted: Vec<f64>, bits: i64): Vec<f64>

A codebook of 2^bits values for the samples (sorted): Lloyd's algorithm in one dimension, where each value's samples are a run of the sorted list, started evenly spaced over the samples' range. (Started at the quantiles, the tails keep wide cells: at 8 bits the error of normal samples is 1.9 times the even start's.)

Effects: fail.

`packed`

gizlang

function packed(rows: Vec<f32>, at: i64, count: i64, k: i64, book: Vec<f64>, th: Vec<f32>, bits: i64, d: Vec<f32>): Packed

Quantizes 16 rows of k inputs (rows[at..], row after row; missing rows are zero) to book: each row's blocks rotated, then for each block of 128 a scale found by alternating the nearest codes and the least-squares scale.

Effects: fail.

`densePacked`

gizlang

function densePacked(rows: Vec<f32>, at: i64, count: i64, k: i64, f: i64, d: Vec<f32>): Packed

16 rows of k inputs (as packed) in dense format f: each row's blocks rotated, then with F8 each block scaled so its largest value is 448.

Effects: fail.

`isK`

gizlang

function isK(f: i64): bool

Whether f is a K format.

`kFormatOf`

gizlang

function kFormatOf(dtype: string): Option<i64>

The K format that stores GGUF block type dtype as it is (Q4_K, Q6_K), or none.

`kSub`

gizlang

function kSub(f: i64): i64

The inputs that share a sub-scale in K format f.

`kPacked`

gizlang

function kPacked(data: bytes, count: i64, k: i64, f: i64): KPacked

16 rows of k inputs from data, the GGUF blocks of count rows (row after row; missing rows are zero), in K format f.

Effects: fail.

`kValue`

gizlang

function kValue(f: i64, k: i64, scales: bytes, subs: Vec<f32>, rb: i64, row: i64, col: i64, q: i64): f64

Weight row (0 to 15) of row block rb, input col, of a linear of k inputs in K format f, from its code q, the linear's scales and its subs as numbers, on the host.

Effects: fail.

`regularRotation`

gizlang

function regularRotation(gpu: Gpu): Tensor

The regular Hadamard rotation as a matrix ([256, 256] BF16 on gpu) of Kernel.hadamard: R[j][i] = regularSign(i) (-1)^popcount(i & j) / 16 (signs, then Sylvester's order).

Effects: fail, read, write, console, io.

Methods

`W4.hostWeights`

gizlang

function W4.hostWeights(self: W4, w: Tensor): Tensor

The INT8 weights of W4 codes w ([out, in / 2], U8 values on the host) by this decoding, on the host, as F32 ([out, in]), each input o at its place after the regular rotation (K.regularSource, signed by K.regularSign): the order int8Rows rotates the inputs in, so the weights take x as int8Rows gives it. decodeInto on a GPU.

Effects: fail.

`W4.decodeInto`

gizlang

function W4.decodeInto(self: W4, w: Tensor, out: Tensor): void

The INT8 weights of W4 codes w ([out, in / 2]) by this decoding, into out ([out, in] I8).

Effects: fail, read, write, console, io.

Constants

`GROUP`

gizlang

const GROUP = 128

The inputs of a linear rotate and share a scale in blocks of this many.

`ROTATED`

gizlang

const ROTATED = 256

The values a regular Hadamard rotation of W4's inputs takes at once (ConvRot's group; netops int8Rows).

`BF16`

gizlang

const BF16 = 16

The format of dense bfloat16 values (see Formats); formats 2 to 8 are codebooks of that many bits.

`F16`

gizlang

const F16 = 17

The format of dense IEEE half-precision values.

`F8`

gizlang

const F8 = 18

The format of FP8 E4M3 values with block scales.

`K4`

gizlang

const K4 = 20

The format of GGUF's Q4_K blocks (see K formats).

`K6`

gizlang

const K6 = 22

The format of GGUF's Q6_K blocks (see K formats).