GizStack
View .mdOpen in ChatGPTOpen in Claude

gizlang:compress

DEFLATE (RFC 1951), gzip (RFC 1952) and CRC checksums, written in Gizlang. deflate finds repeats greedily in a two-way hash table of 4-byte prefixes over a 32 KB window (tokenize), counts the symbols, and writes one block with Huffman codes built for them (dynamic) or DEFLATE's fixed codes, whichever is shorter. Both are canonical codes from lengths, built by the same functions. The tables are constant Vec literals. The checksums run in the runtime (gizlang_crc32_update), on the CPU's CRC instructions.

gizlang
import * as Compress from "gizlang:compress";

Types

Crc32

gizlang
type Crc32 = { state: u64; }

The state of a running CRC-32.

  • state: u64

Crc32c

gizlang
type Crc32c = { state: u64; }

Castagnoli CRC-32C uses a different polynomial from gzip's IEEE CRC-32: the SSE4.2 CRC32 instruction where the CPU has it, else slicing-by-8.

  • state: u64

Deflater

gizlang
type Deflater

A streaming DEFLATE encoder in one framing (RAW, ZLIB or GZIP): write input, flush when a reader must see it now (SYNC_FLUSH ends the block and aligns, as zlib does), finish once at the end, and take the output with read. Each block is coded on its own (block), so a flush costs the repeats across it. Level 0 stores the data.

Opaque: create and use it through this module's functions.

DecodeError

gizlang
type DecodeError = { code: i64; offset: i64; message: string; }

Why a decoder stopped: code is one of the constants below, offset is the input byte where the decoder found the problem, message tells what it is.

  • code: i64
  • offset: i64
  • message: string

Inflater

gizlang
type Inflater

A streaming DEFLATE decoder: give it input with write and close, and take the output with read; or call feed for each piece and finish at the end.

Opaque: create and use it through this module's functions.

Xxh64

gizlang
type Xxh64

XXH64 over data given in pieces, as zstd checks a frame: four lanes of 8 bytes, and the bytes of the last partial stripe waiting in pending.

Opaque: create and use it through this module's functions.

ZstdDecoder

gizlang
type ZstdDecoder

A streaming Zstandard decoder: give it input with write and close, and take the output with read; or call feed for each piece and finish at the end. It decodes each frame of the input, and skips skippable frames.

Opaque: create and use it through this module's functions.

TarMember

gizlang
type TarMember = { path: string; kind: string; mode: i64; data: bytes; link: string; }

One archive member. kind is "file", "directory", "symlink", "hardlink" or "other" (devices and FIFOs); link is the target of a link.

  • path: string
  • kind: string
  • mode: i64
  • data: bytes
  • link: string

BrotliEncoder

gizlang
type BrotliEncoder

A Brotli encoder: write(data) takes input, operate(op) runs PROCESS/FLUSH/FINISH, read() gives what came out.

Opaque: create and use it through this module's functions.

BrotliDecoder

gizlang
type BrotliDecoder

A Brotli decoder: write(data) decodes what it can; an error is libbrotli's message.

Opaque: create and use it through this module's functions.

Alp

gizlang
type Alp = { exponent: i64; digits: Vec<i64>; at: Vec<i64>; exceptions: Vec<f64>; }

A vector of doubles as ALP integers: digits[i] / 10^exponent, except at the positions at, whose values are exceptions.

  • exponent: i64
  • digits: Vec<i64>
  • at: Vec<i64>
  • exceptions: Vec<f64>

ZipWriter

gizlang
type ZipWriter

A zip archive, written in order: the caller writes what each call returns. An entry comes whole (file: its bytes stored or deflated, its CRC and sizes in its header) or as it streams (entry, then data for each piece of its bytes, which the caller writes as they are, then a data descriptor). A streamed entry is stored: most bytes people keep are compressed already, and storing keeps the archive at disk speed. Names are UTF-8 (flag bit 11 when a name is not ASCII). A modified time goes as a DOS time and as UTC seconds (extra 0x5455); modified 0 writes no time (1980-01-01 and no extra field), as the first entry of an OCF container needs. ZIP64 records come in only where a size, an offset or the count needs them. finish writes the central directory.

Opaque: create and use it through this module's functions.

ZipEntry

gizlang
type ZipEntry = { name: string; size: i64; stored: i64; method: u64; crc: u64; modified: i64; offset: i64; }

One entry of a zip archive, as its central directory records it: NAME (a folder's ends in "/"), SIZE bytes, STORED of them in the archive by METHOD (0 stored, 8 deflated), their CRC, MODIFIED (ms since 1970: extra 0x5455's UTC seconds, else the DOS time read as UTC, 0 for none) and OFFSET, where its local header is.

  • name: string
  • size: i64
  • stored: i64
  • method: u64
  • crc: u64
  • modified: i64
  • offset: i64

Functions

crc32

gizlang
function crc32(data: bytes): u64

The CRC-32 of data, as gzip and zip record it.

Effects: fail.

newCrc32

gizlang
function newCrc32(): Crc32

A CRC-32 checksum (the one of zlib, gzip and zip) of no data.

crc32Combine

gizlang
function crc32Combine(first: u64, second: u64, secondBytes: i64): u64 requires secondBytes> = 0 { let factor = CRC32_TOP_BIT

CRC-32 of concatenated regions from their digests and the second length, without joining or rereading their buffers.

Effects: fail.

newCrc32c

gizlang
function newCrc32c(): Crc32c

A CRC-32C (Castagnoli) checksum of no data.

crc32c

gizlang
function crc32c(data: bytes): u64

The CRC-32C checksum of DATA.

Effects: fail.

crc16Xmodem

gizlang
function crc16Xmodem(data: bytes): u64

CRC-16/XMODEM: polynomial 0x1021, initial zero, no reflection or final xor.

Effects: fail.

lz4

gizlang
function lz4(data: bytes): bytes

DATA as one LZ4 block (the LZ4 block format, no frame): greedy matches of 4 or more bytes within 64 KB, found in one table of 4-byte prefixes (hash4, common). It is fast, not small: for bytes that are written and read often. A run of misses makes the search step longer, so data that does not compress costs little.

Effects: fail.

unlz4

gizlang
function unlz4(block: bytes, limit: i64): bytes throws DecodeError requires limit> = 0 { let out = b""

The data of the LZ4 block BLOCK, at most LIMIT bytes of output.

Effects: fail.

deflate

gizlang
function deflate(data: bytes): bytes

data as one DEFLATE stream (no header): a single final block.

Effects: fail.

gzip

gizlang
function gzip(data: bytes): bytes

data as a gzip member: header, DEFLATE stream, CRC-32 and length.

Effects: fail.

newDeflater

gizlang
function newDeflater(framing: i64, level: i64): Deflater requires framing> = RAW && framing <= GZIP

A Deflate compressor at LEVEL (0 to 9) with FRAMING: RAW, ZLIB or GZIP.

Effects: fail.

adler32

gizlang
function adler32(data: bytes): u64

The Adler-32 checksum of data (RFC 1950), as zlib records it.

Effects: fail.

newInflater

gizlang
function newInflater(framing: i64, limit: i64): Inflater requires framing> = RAW && framing <= GZIP requires limit> = 0 { let state = I_HEADER

A decoder for one framing (RAW, ZLIB or GZIP) that refuses more than limit bytes of output. GZIP decodes each member of a multi-member file.

Effects: fail.

inflater

gizlang
function inflater(framing: i64): Inflater requires framing> = RAW && framing <= GZIP

A decoder of framing RAW, ZLIB or GZIP with no output limit, for feed and finish.

Effects: fail.

inflate

gizlang
function inflate(data: bytes, framing: i64, limit: i64): bytes throws DecodeError requires framing> = RAW && framing <= GZIP requires limit> = 0 { let d = new Inflater(framing, limit)

The DEFLATE stream data decoded (framing RAW, ZLIB or GZIP), at most limit bytes of output.

Effects: fail.

gunzip

gizlang
function gunzip(data: bytes, limit: i64): bytes throws DecodeError requires limit> = 0 { return inflate(data, GZIP, limit)

The gzip file data decoded (every member), at most limit bytes of output.

Effects: fail.

newXxh64

gizlang
function newXxh64(seed: u64): Xxh64

A new XXH64 hash state with SEED.

xxh64

gizlang
function xxh64(data: bytes, seed: u64): u64

The XXH64 of data with a seed.

Effects: fail.

newZstdDecoder

gizlang
function newZstdDecoder(limit: i64, windowLimit: i64): ZstdDecoder requires limit> = 0 && windowLimit> = 0 { return { _limit: limit, _windowLimit: windowLimit, _input: b"", _at: 0, _base: 0, _ended: false, _end: 0, _state: Z_MAGIC, _skip: 0, _frames: 0, _window: 0, _fcs: -1, _frameOut: 0, _hasChecksum: false, _blockMax: 0, _buf: b"", _pos: 0, _start: 0, _floor: 0, _produced: 0, _hash: new Xxh64(0u), _lits: filled(ZSTD_BLOCK + 64, 0u), _tables: noEntropy(), _dictTables: noEntropy(), _dict: b"", _dictId: 0, _llDefault: fseTable(literalLengthDefault(), 6, FSE_LITERALS), _mlDefault: fseTable(matchLengthDefault(), 6, FSE_MATCHES), _ofDefault: fseTable(offsetDefault(), 5, FSE_OFFSETS), _error: 0, _errorAt: 0, _message: "", }

A decoder that refuses more than limit bytes of output and frames with a window over windowLimit bytes (ZSTD_WINDOW_LIMIT is the zstd tool's). Its memory is about twice the window of the frame, and no more than the limit.

Effects: fail.

zstdDecoder

gizlang
function zstdDecoder(): ZstdDecoder

A decoder with no output limit and windows up to ZSTD_WINDOW_LIMIT, for feed and finish.

Effects: fail.

unzstd

gizlang
function unzstd(data: bytes, limit: i64): bytes throws DecodeError requires limit> = 0 { let d = new ZstdDecoder(limit, ZSTD_WINDOW_LIMIT)

The Zstandard data decoded (every frame), at most limit bytes of output, with windows up to ZSTD_WINDOW_LIMIT.

Effects: fail.

unzstdWith

gizlang
function unzstdWith(data: bytes, dictionary: bytes, limit: i64): bytes throws DecodeError requires limit> = 0 { let d = new ZstdDecoder(limit, ZSTD_WINDOW_LIMIT)

unzstd with a dictionary (see useDictionary).

Effects: fail.

untar

gizlang
function untar(archive: bytes, limit: i64): Vec<TarMember> throws DecodeError requires limit> = 0 { let members: Vec<TarMember> = []

Every member of archive in order, with pax and GNU long-name headers applied to the member after them. It reads the archive as given: the caller decides which members to keep. It fails (CORRUPT) on a header whose checksum is wrong or whose data runs past the archive, and (TOO_LARGE) when the members hold more than limit bytes (their bodies and the pax and long-name records).

Effects: fail.

tarPadding

gizlang
function tarPadding(size: i64): i64

The zero bytes after a member body of size bytes, to the next 512-byte block.

Effects: fail.

tarHeader

gizlang
function tarHeader(path: string, mode: i64, size: i64, link: string): bytes

The header blocks of one member fixed by its fields alone (owner 0, time 0): a pax header when PATH or LINK is longer than 100 bytes, then the ustar block. A member with a LINK is a hard link with no body; otherwise it is a file of SIZE bytes, whose body and tarPadding(size) zero bytes follow. Two archives of the same members are the same bytes.

Effects: fail.

newBrotliEncoder

gizlang
function newBrotliEncoder(): BrotliEncoder

A Brotli compressor with the default parameters.

brotli

gizlang
function brotli(data: bytes, quality: i64): Result<bytes, string>

DATA in one Brotli stream at QUALITY with a 4 MB window (22).

Effects: fail.

newBrotliDecoder

gizlang
function newBrotliDecoder(): BrotliDecoder

A Brotli decompressor with the default parameters.

fsstEncode

gizlang
function fsstEncode(data: bytes): bytes

data compressed by FSST: its length (u64), the symbol table (a count, then each symbol's length and bytes) and the codes. Sizes and contents of the input are kept exactly; fsstDecode reads it back.

Effects: fail.

fsstDecode

gizlang
function fsstDecode(data: bytes): bytes throws string

The bytes of an FSST-compressed block (its symbol table, then the codes).

Effects: fail.

fsstDecodeRanges

gizlang
function fsstDecodeRanges(data: bytes, from: Vec<i64>, to: Vec<i64>): bytes throws string

The bytes [FROM[k], TO[k]) of fsstDecode(DATA) for each k, joined: FROM ascending, the ranges apart. The codes before a range are only counted, and decoding stops after the last range: a top-k's few rows of a block of 100K strings (ClickBench Q23).

Effects: fail.

alpEncode

gizlang
function alpEncode(v: Vec<f64>): Alp

The values of V in ALP form: integers at a decimal scale, with the values that do not fit kept apart.

Effects: fail.

alpDecode

gizlang
function alpDecode(a: Alp): Vec<f64> throws string

The doubles of a, or an error for an exponent or exception position out of range.

Effects: fail.

newZipWriter

gizlang
function newZipWriter(): ZipWriter

A writer of a zip archive in memory, with no entries.

zipEntries

gizlang
function zipEntries(size: i64, read: (at: i64, count: i64) => Result<bytes, string>, most: i64): Vec<ZipEntry> throws DecodeError

The first MOST entries of the zip archive of SIZE bytes that READ(at, count) reads, in their central directory's order, ZIP64 included. It reads the end records and the central directory, never an entry's bytes. A name that is not UTF-8 (an old archive's code page) shows each byte past ASCII as U+FFFD.

Effects: fail, read, write, console, io.

unzipEntry

gizlang
function unzipEntry(e: ZipEntry, read: (at: i64, count: i64) => Result<bytes, string>, limit: i64): bytes throws DecodeError requires limit> = 0 { if (e.size> limit) { return Result.err(DecodeError(TOO_LARGE, e.offset, `zip: ${ e.name } has ${ e.size } bytes, over the limit of ${ limit }`))

The bytes of entry E of the archive that READ(at, count) reads, checked against its CRC: TOO_LARGE past LIMIT bytes, UNSUPPORTED when it is encrypted or packed by a method other than stored or deflated.

Effects: fail, read, write, console, io.

Methods

Crc32.update

gizlang
function Crc32.update(self: mut Crc32, data: bytes): void

Adds DATA to the checksum.

Effects: fail.

Crc32.digest

gizlang
function Crc32.digest(self: Crc32): u64

The checksum of the data added so far.

Effects: fail.

Crc32c.update

gizlang
function Crc32c.update(self: mut Crc32c, data: bytes): void

Chunk boundaries never alter the checksum; data can remain in its current buffers.

Effects: fail.

Crc32c.digest

gizlang
function Crc32c.digest(self: Crc32c): u64

The checksum of the data added so far.

Effects: fail.

Deflater.level

gizlang
function Deflater.level(self: Deflater): i64

The compression level (0 to 9).

Deflater.setLevel

gizlang
function Deflater.setLevel(self: mut Deflater, level: i64): void

Sets the compression level (0 to 9) for the input that follows.

Deflater.finished

gizlang
function Deflater.finished(self: Deflater): bool

Whether the stream has ended (a FINISH flush is done).

Deflater.read

gizlang
function Deflater.read(self: mut Deflater): bytes

The output produced so far, taken.

Deflater.write

gizlang
function Deflater.write(self: mut Deflater, input: bytes): void requires!self._finished

Add input; whole blocks of it are coded as they fill.

Effects: fail.

Deflater.flush

gizlang
function Deflater.flush(self: mut Deflater, kind: i64): void

The pending input coded (KIND other than NO_FLUSH); SYNC_FLUSH and FULL_FLUSH add an empty stored block so the output ends on a byte with all the input decodable; FINISH ends the stream with its trailer.

Effects: fail.

Inflater.write

gizlang
function Inflater.write(self: mut Inflater, input: bytes): void requires!self._ended

Add input. The decoder keeps only the input that it has not decoded.

Effects: fail.

Inflater.close

gizlang
function Inflater.close(self: mut Inflater): void

Mark the end of the input. After it, a stream that has not ended is TRUNCATED.

Effects: fail.

Inflater.done

gizlang
function Inflater.done(self: Inflater): bool

Whether the stream has ended and all its output was read.

Inflater.read

gizlang
function Inflater.read(self: mut Inflater): bytes throws DecodeError

The next output, at most 256 KiB. Empty when the decoder needs input (write more, or close) or when it is done. An error stays: each later read returns it again.

Effects: fail.

Inflater.feed

gizlang
function Inflater.feed(self: mut Inflater, input: bytes): bytes throws string

Write input and return all the output that it makes now.

Effects: fail.

Inflater.finish

gizlang
function Inflater.finish(self: mut Inflater): bytes throws string

End the input and return the rest of the output. A stream that has not ended is an error.

Effects: fail.

Xxh64.update

gizlang
function Xxh64.update(self: mut Xxh64, data: bytes, from: i64, to: i64): void requires from> = 0 && from <= to && to <= data.length { self._total + = to - from

Add the bytes from..to of data.

Effects: fail.

Xxh64.digest

gizlang
function Xxh64.digest(self: Xxh64): u64

The XXH64 of the bytes so far.

Effects: fail.

ZstdDecoder.useDictionary

gizlang
function ZstdDecoder.useDictionary(self: mut ZstdDecoder, dictionary: bytes): i64 throws DecodeError

Use a dictionary for the frames that follow: a zstd dictionary (its entropy tables, repeat offsets and content) or, without the dictionary magic, raw content. Its ID is what a frame names (0 for raw content).

Effects: fail.

ZstdDecoder.write

gizlang
function ZstdDecoder.write(self: mut ZstdDecoder, input: bytes): void requires!self._ended

Add input. The decoder keeps only the input that it has not decoded.

Effects: fail.

ZstdDecoder.close

gizlang
function ZstdDecoder.close(self: mut ZstdDecoder): void

Mark the end of the input. After it, a frame that has not ended is TRUNCATED.

Effects: fail.

ZstdDecoder.done

gizlang
function ZstdDecoder.done(self: ZstdDecoder): bool

Whether the input has ended after a whole frame, and all output was read.

ZstdDecoder.read

gizlang
function ZstdDecoder.read(self: mut ZstdDecoder): bytes throws DecodeError

The next output, at most 1 MiB. Empty when the decoder needs input (write more, or close) or when it is done. An error stays: each later read returns it again.

Effects: fail.

ZstdDecoder.feed

gizlang
function ZstdDecoder.feed(self: mut ZstdDecoder, input: bytes): bytes throws string

Write input and return all the output that it makes now.

Effects: fail.

ZstdDecoder.finish

gizlang
function ZstdDecoder.finish(self: mut ZstdDecoder): bytes throws string

End the input and return the rest of the output. A frame that has not ended is an error.

Effects: fail.

BrotliEncoder.setParameter

gizlang
function BrotliEncoder.setParameter(self: mut BrotliEncoder, param: i64, value: i64): bool

Sets the parameter PARAM (BROTLI_QUALITY or BROTLI_WINDOW) to VALUE; false when libbrotli refuses it.

BrotliEncoder.finished

gizlang
function BrotliEncoder.finished(self: BrotliEncoder): bool

Whether the stream has ended.

BrotliEncoder.write

gizlang
function BrotliEncoder.write(self: mut BrotliEncoder, input: bytes, op: i64): bool

INPUT compressed with OP; the output joins the pending bytes. False when libbrotli refused.

Effects: fail.

BrotliEncoder.read

gizlang
function BrotliEncoder.read(self: mut BrotliEncoder): bytes

The output produced so far, taken.

BrotliEncoder.close

gizlang
function BrotliEncoder.close(self: mut BrotliEncoder): void

Frees the compressor. It does nothing after the first call.

BrotliDecoder.errorCode

gizlang
function BrotliDecoder.errorCode(self: BrotliDecoder): i64

libbrotli's BrotliDecoderErrorCode of the last failed write (negative), 0 when none.

BrotliDecoder.setParameter

gizlang
function BrotliDecoder.setParameter(self: mut BrotliDecoder, param: i64, value: i64): bool

Sets the decoder parameter PARAM to VALUE; false when libbrotli refuses it.

BrotliDecoder.done

gizlang
function BrotliDecoder.done(self: BrotliDecoder): bool

Whether the stream has ended.

BrotliDecoder.write

gizlang
function BrotliDecoder.write(self: mut BrotliDecoder, input: bytes): bool throws string

Decompresses INPUT; the output joins the pending bytes. It fails on bad data or after close.

Effects: fail.

BrotliDecoder.read

gizlang
function BrotliDecoder.read(self: mut BrotliDecoder): bytes

The output produced so far, taken.

BrotliDecoder.close

gizlang
function BrotliDecoder.close(self: mut BrotliDecoder): void

Frees the decompressor. It does nothing after the first call.

ZipWriter.entry

gizlang
function ZipWriter.entry(self: mut ZipWriter, name: string, modified: i64): bytes

The local header of a streamed entry NAME (a folder ends in "/"), modified at MODIFIED (ms since 1970, 0: no time); it ends the streamed entry before.

Effects: fail.

ZipWriter.data

gizlang
function ZipWriter.data(self: mut ZipWriter, data: bytes): void requires self._open && !self._folder

Counts DATA, the next bytes of the streamed entry, which the caller writes as they are.

Effects: fail.

ZipWriter.file

gizlang
function ZipWriter.file(self: mut ZipWriter, name: string, modified: i64, data: bytes, deflated: bool): bytes requires data.length <4294967295 { let out = self.closeEntry()

Entry NAME whole: its header and DATA, deflated when DEFLATED and that is smaller, modified at MODIFIED (ms since 1970, 0: no time). It ends the streamed entry before.

Effects: fail.

ZipWriter.finish

gizlang
function ZipWriter.finish(self: mut ZipWriter): bytes

The end of the archive: a streamed entry's descriptor, the central directory and the end records.

Effects: fail.

Constants

NO_FLUSH

gizlang
const NO_FLUSH = 0

The flush mode no flush, 0.

PARTIAL_FLUSH

gizlang
const PARTIAL_FLUSH = 1

The flush mode partial flush, 1.

SYNC_FLUSH

gizlang
const SYNC_FLUSH = 2

The flush mode sync flush, 2.

FULL_FLUSH

gizlang
const FULL_FLUSH = 3

The flush mode full flush, 3.

FINISH

gizlang
const FINISH = 4

The flush mode finish, 4.

BLOCK_FLUSH

gizlang
const BLOCK_FLUSH = 5

The flush mode block flush, 5.

CORRUPT

gizlang
const CORRUPT = 1

The input is not a valid stream.

TRUNCATED

gizlang
const TRUNCATED = 2

The input ends before the stream does.

TOO_LARGE

gizlang
const TOO_LARGE = 3

The output, or the window that a frame asks for, is over the limit of the caller.

CHECKSUM

gizlang
const CHECKSUM = 4

A checksum or a recorded length does not agree with the output.

UNSUPPORTED

gizlang
const UNSUPPORTED = 5

The stream uses a feature that the decoder does not have (a zlib preset dictionary).

NEEDS_DICTIONARY

gizlang
const NEEDS_DICTIONARY = 6

The frame needs a dictionary that the decoder does not have.

RAW

gizlang
const RAW = 0

DEFLATE framings: no framing (RFC 1951), zlib (RFC 1950) and gzip (RFC 1952).

ZLIB

gizlang
const ZLIB = 1

The zlib framing (RFC 1950).

GZIP

gizlang
const GZIP = 2

The gzip framing (RFC 1952).

ZSTD_WINDOW_LIMIT

gizlang
const ZSTD_WINDOW_LIMIT = 134217728

The default window limit, as the zstd tool's (2^27).

BROTLI_PROCESS

gizlang
const BROTLI_PROCESS = 0

BrotliEncoderOperation.

BROTLI_FLUSH

gizlang
const BROTLI_FLUSH = 1

The flush mode brotli flush, 1.

BROTLI_FINISH

gizlang
const BROTLI_FINISH = 2

The flush mode brotli finish, 2.

BROTLI_QUALITY

gizlang
const BROTLI_QUALITY = 1

BrotliEncoderParameter: the quality (0 to 11) and the window (log2 of its size, 10 to 24).

BROTLI_WINDOW

gizlang
const BROTLI_WINDOW = 2

The parameter of the window size, as log2 of its bytes (10 to 24).