gizlang:compress
DEFLATE (RFC 1951), gzip (RFC 1952) and CRC checksums, written in Gizlang. deflate finds repeats greedily in a two-way hash table of 4-byte prefixes over a 32 KB window (tokenize), counts the symbols, and writes one block with Huffman codes built for them (dynamic) or DEFLATE's fixed codes, whichever is shorter. Both are canonical codes from lengths, built by the same functions. The tables are constant Vec literals. The checksums run in the runtime (gizlang_crc32_update), on the CPU's CRC instructions.
import * as Compress from "gizlang:compress";Types
Crc32
type Crc32 = { state: u64; }The state of a running CRC-32.
state:u64
Crc32c
type Crc32c = { state: u64; }Castagnoli CRC-32C uses a different polynomial from gzip's IEEE CRC-32: the SSE4.2 CRC32 instruction where the CPU has it, else slicing-by-8.
state:u64
Deflater
type DeflaterA streaming DEFLATE encoder in one framing (RAW, ZLIB or GZIP): write input, flush when a reader must see it now (SYNC_FLUSH ends the block and aligns, as zlib does), finish once at the end, and take the output with read. Each block is coded on its own (block), so a flush costs the repeats across it. Level 0 stores the data.
Opaque: create and use it through this module's functions.
DecodeError
type DecodeError = { code: i64; offset: i64; message: string; }Why a decoder stopped: code is one of the constants below, offset is the input byte where the decoder found the problem, message tells what it is.
code:i64offset:i64message:string
Inflater
type InflaterA streaming DEFLATE decoder: give it input with write and close, and take the output with read; or call feed for each piece and finish at the end.
Opaque: create and use it through this module's functions.
Xxh64
type Xxh64XXH64 over data given in pieces, as zstd checks a frame: four lanes of 8 bytes, and the bytes of the last partial stripe waiting in pending.
Opaque: create and use it through this module's functions.
ZstdDecoder
type ZstdDecoderA streaming Zstandard decoder: give it input with write and close, and take the output with read; or call feed for each piece and finish at the end. It decodes each frame of the input, and skips skippable frames.
Opaque: create and use it through this module's functions.
TarMember
type TarMember = { path: string; kind: string; mode: i64; data: bytes; link: string; }One archive member. kind is "file", "directory", "symlink", "hardlink" or "other" (devices and FIFOs); link is
the target of a link.
path:stringkind:stringmode:i64data:byteslink:string
BrotliEncoder
type BrotliEncoderA Brotli encoder: write(data) takes input, operate(op) runs PROCESS/FLUSH/FINISH, read() gives what came out.
Opaque: create and use it through this module's functions.
BrotliDecoder
type BrotliDecoderA Brotli decoder: write(data) decodes what it can; an error is libbrotli's message.
Opaque: create and use it through this module's functions.
Alp
type Alp = { exponent: i64; digits: Vec<i64>; at: Vec<i64>; exceptions: Vec<f64>; }A vector of doubles as ALP integers: digits[i] / 10^exponent, except at the positions at, whose values are
exceptions.
exponent:i64digits:Vec<i64>at:Vec<i64>exceptions:Vec<f64>
ZipWriter
type ZipWriterA zip archive, written in order: the caller writes what each call returns. An entry comes whole (file: its bytes
stored or deflated, its CRC and sizes in its header) or as it streams (entry, then data for each piece of its
bytes, which the caller writes as they are, then a data descriptor). A streamed entry is stored: most bytes people
keep are compressed already, and storing keeps the archive at disk speed. Names are UTF-8 (flag bit 11 when a name is
not ASCII). A modified time goes as a DOS time and as UTC seconds (extra 0x5455); modified 0 writes no time
(1980-01-01 and no extra field), as the first entry of an OCF container needs. ZIP64 records come in only where a
size, an offset or the count needs them. finish writes the central directory.
Opaque: create and use it through this module's functions.
ZipEntry
type ZipEntry = { name: string; size: i64; stored: i64; method: u64; crc: u64; modified: i64; offset: i64; }One entry of a zip archive, as its central directory records it: NAME (a folder's ends in "/"), SIZE bytes, STORED of them in the archive by METHOD (0 stored, 8 deflated), their CRC, MODIFIED (ms since 1970: extra 0x5455's UTC seconds, else the DOS time read as UTC, 0 for none) and OFFSET, where its local header is.
name:stringsize:i64stored:i64method:u64crc:u64modified:i64offset:i64
Functions
crc32
function crc32(data: bytes): u64The CRC-32 of data, as gzip and zip record it.
Effects: fail.
newCrc32
function newCrc32(): Crc32A CRC-32 checksum (the one of zlib, gzip and zip) of no data.
crc32Combine
function crc32Combine(first: u64, second: u64, secondBytes: i64): u64 requires secondBytes> = 0 { let factor = CRC32_TOP_BITCRC-32 of concatenated regions from their digests and the second length, without joining or rereading their buffers.
Effects: fail.
newCrc32c
function newCrc32c(): Crc32cA CRC-32C (Castagnoli) checksum of no data.
crc32c
function crc32c(data: bytes): u64The CRC-32C checksum of DATA.
Effects: fail.
crc16Xmodem
function crc16Xmodem(data: bytes): u64CRC-16/XMODEM: polynomial 0x1021, initial zero, no reflection or final xor.
Effects: fail.
lz4
function lz4(data: bytes): bytesDATA as one LZ4 block (the LZ4 block format, no frame): greedy matches of 4 or more bytes within 64 KB, found in one table of 4-byte prefixes (hash4, common). It is fast, not small: for bytes that are written and read often. A run of misses makes the search step longer, so data that does not compress costs little.
Effects: fail.
unlz4
function unlz4(block: bytes, limit: i64): bytes throws DecodeError requires limit> = 0 { let out = b""The data of the LZ4 block BLOCK, at most LIMIT bytes of output.
Effects: fail.
deflate
function deflate(data: bytes): bytesdata as one DEFLATE stream (no header): a single final block.
Effects: fail.
gzip
function gzip(data: bytes): bytesdata as a gzip member: header, DEFLATE stream, CRC-32 and length.
Effects: fail.
newDeflater
function newDeflater(framing: i64, level: i64): Deflater requires framing> = RAW && framing <= GZIPA Deflate compressor at LEVEL (0 to 9) with FRAMING: RAW, ZLIB or GZIP.
Effects: fail.
adler32
function adler32(data: bytes): u64The Adler-32 checksum of data (RFC 1950), as zlib records it.
Effects: fail.
newInflater
function newInflater(framing: i64, limit: i64): Inflater requires framing> = RAW && framing <= GZIP requires limit> = 0 { let state = I_HEADERA decoder for one framing (RAW, ZLIB or GZIP) that refuses more than limit bytes of output. GZIP decodes each member of a multi-member file.
Effects: fail.
inflater
function inflater(framing: i64): Inflater requires framing> = RAW && framing <= GZIPA decoder of framing RAW, ZLIB or GZIP with no output limit, for feed and finish.
Effects: fail.
inflate
function inflate(data: bytes, framing: i64, limit: i64): bytes throws DecodeError requires framing> = RAW && framing <= GZIP requires limit> = 0 { let d = new Inflater(framing, limit)The DEFLATE stream data decoded (framing RAW, ZLIB or GZIP), at most limit bytes of output.
Effects: fail.
gunzip
function gunzip(data: bytes, limit: i64): bytes throws DecodeError requires limit> = 0 { return inflate(data, GZIP, limit)The gzip file data decoded (every member), at most limit bytes of output.
Effects: fail.
newXxh64
function newXxh64(seed: u64): Xxh64A new XXH64 hash state with SEED.
xxh64
function xxh64(data: bytes, seed: u64): u64The XXH64 of data with a seed.
Effects: fail.
newZstdDecoder
function newZstdDecoder(limit: i64, windowLimit: i64): ZstdDecoder requires limit> = 0 && windowLimit> = 0 { return { _limit: limit, _windowLimit: windowLimit, _input: b"", _at: 0, _base: 0, _ended: false, _end: 0, _state: Z_MAGIC, _skip: 0, _frames: 0, _window: 0, _fcs: -1, _frameOut: 0, _hasChecksum: false, _blockMax: 0, _buf: b"", _pos: 0, _start: 0, _floor: 0, _produced: 0, _hash: new Xxh64(0u), _lits: filled(ZSTD_BLOCK + 64, 0u), _tables: noEntropy(), _dictTables: noEntropy(), _dict: b"", _dictId: 0, _llDefault: fseTable(literalLengthDefault(), 6, FSE_LITERALS), _mlDefault: fseTable(matchLengthDefault(), 6, FSE_MATCHES), _ofDefault: fseTable(offsetDefault(), 5, FSE_OFFSETS), _error: 0, _errorAt: 0, _message: "", }A decoder that refuses more than limit bytes of output and frames with a window over windowLimit bytes (ZSTD_WINDOW_LIMIT is the zstd tool's). Its memory is about twice the window of the frame, and no more than the limit.
Effects: fail.
zstdDecoder
function zstdDecoder(): ZstdDecoderA decoder with no output limit and windows up to ZSTD_WINDOW_LIMIT, for feed and finish.
Effects: fail.
unzstd
function unzstd(data: bytes, limit: i64): bytes throws DecodeError requires limit> = 0 { let d = new ZstdDecoder(limit, ZSTD_WINDOW_LIMIT)The Zstandard data decoded (every frame), at most limit bytes of output, with windows up to ZSTD_WINDOW_LIMIT.
Effects: fail.
unzstdWith
function unzstdWith(data: bytes, dictionary: bytes, limit: i64): bytes throws DecodeError requires limit> = 0 { let d = new ZstdDecoder(limit, ZSTD_WINDOW_LIMIT)unzstd with a dictionary (see useDictionary).
Effects: fail.
untar
function untar(archive: bytes, limit: i64): Vec<TarMember> throws DecodeError requires limit> = 0 { let members: Vec<TarMember> = []Every member of archive in order, with pax and GNU long-name headers applied to the member after them. It reads the
archive as given: the caller decides which members to keep. It fails (CORRUPT) on a header whose checksum is wrong or
whose data runs past the archive, and (TOO_LARGE) when the members hold more than limit bytes (their bodies and
the pax and long-name records).
Effects: fail.
tarPadding
function tarPadding(size: i64): i64The zero bytes after a member body of size bytes, to the next 512-byte block.
Effects: fail.
tarHeader
function tarHeader(path: string, mode: i64, size: i64, link: string): bytesThe header blocks of one member fixed by its fields alone (owner 0, time 0): a pax header when PATH or LINK is longer
than 100 bytes, then the ustar block. A member with a LINK is a hard link with no body; otherwise it is a file of SIZE
bytes, whose body and tarPadding(size) zero bytes follow. Two archives of the same members are the same bytes.
Effects: fail.
newBrotliEncoder
function newBrotliEncoder(): BrotliEncoderA Brotli compressor with the default parameters.
brotli
function brotli(data: bytes, quality: i64): Result<bytes, string>DATA in one Brotli stream at QUALITY with a 4 MB window (22).
Effects: fail.
newBrotliDecoder
function newBrotliDecoder(): BrotliDecoderA Brotli decompressor with the default parameters.
fsstEncode
function fsstEncode(data: bytes): bytesdata compressed by FSST: its length (u64), the symbol table (a count, then each symbol's length and bytes) and the
codes. Sizes and contents of the input are kept exactly; fsstDecode reads it back.
Effects: fail.
fsstDecode
function fsstDecode(data: bytes): bytes throws stringThe bytes of an FSST-compressed block (its symbol table, then the codes).
Effects: fail.
fsstDecodeRanges
function fsstDecodeRanges(data: bytes, from: Vec<i64>, to: Vec<i64>): bytes throws stringThe bytes [FROM[k], TO[k]) of fsstDecode(DATA) for each k, joined: FROM ascending, the ranges apart. The codes before a range are only counted, and decoding stops after the last range: a top-k's few rows of a block of 100K strings (ClickBench Q23).
Effects: fail.
alpEncode
function alpEncode(v: Vec<f64>): AlpThe values of V in ALP form: integers at a decimal scale, with the values that do not fit kept apart.
Effects: fail.
alpDecode
function alpDecode(a: Alp): Vec<f64> throws stringThe doubles of a, or an error for an exponent or exception position out of range.
Effects: fail.
newZipWriter
function newZipWriter(): ZipWriterA writer of a zip archive in memory, with no entries.
zipEntries
function zipEntries(size: i64, read: (at: i64, count: i64) => Result<bytes, string>, most: i64): Vec<ZipEntry> throws DecodeErrorThe first MOST entries of the zip archive of SIZE bytes that READ(at, count) reads, in their central directory's order, ZIP64 included. It reads the end records and the central directory, never an entry's bytes. A name that is not UTF-8 (an old archive's code page) shows each byte past ASCII as U+FFFD.
Effects: fail, read, write, console, io.
unzipEntry
function unzipEntry(e: ZipEntry, read: (at: i64, count: i64) => Result<bytes, string>, limit: i64): bytes throws DecodeError requires limit> = 0 { if (e.size> limit) { return Result.err(DecodeError(TOO_LARGE, e.offset, `zip: ${ e.name } has ${ e.size } bytes, over the limit of ${ limit }`))The bytes of entry E of the archive that READ(at, count) reads, checked against its CRC: TOO_LARGE past LIMIT bytes, UNSUPPORTED when it is encrypted or packed by a method other than stored or deflated.
Effects: fail, read, write, console, io.
Methods
Crc32.update
function Crc32.update(self: mut Crc32, data: bytes): voidAdds DATA to the checksum.
Effects: fail.
Crc32.digest
function Crc32.digest(self: Crc32): u64The checksum of the data added so far.
Effects: fail.
Crc32c.update
function Crc32c.update(self: mut Crc32c, data: bytes): voidChunk boundaries never alter the checksum; data can remain in its current buffers.
Effects: fail.
Crc32c.digest
function Crc32c.digest(self: Crc32c): u64The checksum of the data added so far.
Effects: fail.
Deflater.level
function Deflater.level(self: Deflater): i64The compression level (0 to 9).
Deflater.setLevel
function Deflater.setLevel(self: mut Deflater, level: i64): voidSets the compression level (0 to 9) for the input that follows.
Deflater.finished
function Deflater.finished(self: Deflater): boolWhether the stream has ended (a FINISH flush is done).
Deflater.read
function Deflater.read(self: mut Deflater): bytesThe output produced so far, taken.
Deflater.write
function Deflater.write(self: mut Deflater, input: bytes): void requires!self._finishedAdd input; whole blocks of it are coded as they fill.
Effects: fail.
Deflater.flush
function Deflater.flush(self: mut Deflater, kind: i64): voidThe pending input coded (KIND other than NO_FLUSH); SYNC_FLUSH and FULL_FLUSH add an empty stored block so the output ends on a byte with all the input decodable; FINISH ends the stream with its trailer.
Effects: fail.
Inflater.write
function Inflater.write(self: mut Inflater, input: bytes): void requires!self._endedAdd input. The decoder keeps only the input that it has not decoded.
Effects: fail.
Inflater.close
function Inflater.close(self: mut Inflater): voidMark the end of the input. After it, a stream that has not ended is TRUNCATED.
Effects: fail.
Inflater.done
function Inflater.done(self: Inflater): boolWhether the stream has ended and all its output was read.
Inflater.read
function Inflater.read(self: mut Inflater): bytes throws DecodeErrorThe next output, at most 256 KiB. Empty when the decoder needs input (write more, or close) or when it is done. An error stays: each later read returns it again.
Effects: fail.
Inflater.feed
function Inflater.feed(self: mut Inflater, input: bytes): bytes throws stringWrite input and return all the output that it makes now.
Effects: fail.
Inflater.finish
function Inflater.finish(self: mut Inflater): bytes throws stringEnd the input and return the rest of the output. A stream that has not ended is an error.
Effects: fail.
Xxh64.update
function Xxh64.update(self: mut Xxh64, data: bytes, from: i64, to: i64): void requires from> = 0 && from <= to && to <= data.length { self._total + = to - fromAdd the bytes from..to of data.
Effects: fail.
Xxh64.digest
function Xxh64.digest(self: Xxh64): u64The XXH64 of the bytes so far.
Effects: fail.
ZstdDecoder.useDictionary
function ZstdDecoder.useDictionary(self: mut ZstdDecoder, dictionary: bytes): i64 throws DecodeErrorUse a dictionary for the frames that follow: a zstd dictionary (its entropy tables, repeat offsets and content) or, without the dictionary magic, raw content. Its ID is what a frame names (0 for raw content).
Effects: fail.
ZstdDecoder.write
function ZstdDecoder.write(self: mut ZstdDecoder, input: bytes): void requires!self._endedAdd input. The decoder keeps only the input that it has not decoded.
Effects: fail.
ZstdDecoder.close
function ZstdDecoder.close(self: mut ZstdDecoder): voidMark the end of the input. After it, a frame that has not ended is TRUNCATED.
Effects: fail.
ZstdDecoder.done
function ZstdDecoder.done(self: ZstdDecoder): boolWhether the input has ended after a whole frame, and all output was read.
ZstdDecoder.read
function ZstdDecoder.read(self: mut ZstdDecoder): bytes throws DecodeErrorThe next output, at most 1 MiB. Empty when the decoder needs input (write more, or close) or when it is done. An error stays: each later read returns it again.
Effects: fail.
ZstdDecoder.feed
function ZstdDecoder.feed(self: mut ZstdDecoder, input: bytes): bytes throws stringWrite input and return all the output that it makes now.
Effects: fail.
ZstdDecoder.finish
function ZstdDecoder.finish(self: mut ZstdDecoder): bytes throws stringEnd the input and return the rest of the output. A frame that has not ended is an error.
Effects: fail.
BrotliEncoder.setParameter
function BrotliEncoder.setParameter(self: mut BrotliEncoder, param: i64, value: i64): boolSets the parameter PARAM (BROTLI_QUALITY or BROTLI_WINDOW) to VALUE; false when libbrotli refuses it.
BrotliEncoder.finished
function BrotliEncoder.finished(self: BrotliEncoder): boolWhether the stream has ended.
BrotliEncoder.write
function BrotliEncoder.write(self: mut BrotliEncoder, input: bytes, op: i64): boolINPUT compressed with OP; the output joins the pending bytes. False when libbrotli refused.
Effects: fail.
BrotliEncoder.read
function BrotliEncoder.read(self: mut BrotliEncoder): bytesThe output produced so far, taken.
BrotliEncoder.close
function BrotliEncoder.close(self: mut BrotliEncoder): voidFrees the compressor. It does nothing after the first call.
BrotliDecoder.errorCode
function BrotliDecoder.errorCode(self: BrotliDecoder): i64libbrotli's BrotliDecoderErrorCode of the last failed write (negative), 0 when none.
BrotliDecoder.setParameter
function BrotliDecoder.setParameter(self: mut BrotliDecoder, param: i64, value: i64): boolSets the decoder parameter PARAM to VALUE; false when libbrotli refuses it.
BrotliDecoder.done
function BrotliDecoder.done(self: BrotliDecoder): boolWhether the stream has ended.
BrotliDecoder.write
function BrotliDecoder.write(self: mut BrotliDecoder, input: bytes): bool throws stringDecompresses INPUT; the output joins the pending bytes. It fails on bad data or after close.
Effects: fail.
BrotliDecoder.read
function BrotliDecoder.read(self: mut BrotliDecoder): bytesThe output produced so far, taken.
BrotliDecoder.close
function BrotliDecoder.close(self: mut BrotliDecoder): voidFrees the decompressor. It does nothing after the first call.
ZipWriter.entry
function ZipWriter.entry(self: mut ZipWriter, name: string, modified: i64): bytesThe local header of a streamed entry NAME (a folder ends in "/"), modified at MODIFIED (ms since 1970, 0: no time); it ends the streamed entry before.
Effects: fail.
ZipWriter.data
function ZipWriter.data(self: mut ZipWriter, data: bytes): void requires self._open && !self._folderCounts DATA, the next bytes of the streamed entry, which the caller writes as they are.
Effects: fail.
ZipWriter.file
function ZipWriter.file(self: mut ZipWriter, name: string, modified: i64, data: bytes, deflated: bool): bytes requires data.length <4294967295 { let out = self.closeEntry()Entry NAME whole: its header and DATA, deflated when DEFLATED and that is smaller, modified at MODIFIED (ms since 1970, 0: no time). It ends the streamed entry before.
Effects: fail.
ZipWriter.finish
function ZipWriter.finish(self: mut ZipWriter): bytesThe end of the archive: a streamed entry's descriptor, the central directory and the end records.
Effects: fail.
Constants
NO_FLUSH
const NO_FLUSH = 0The flush mode no flush, 0.
PARTIAL_FLUSH
const PARTIAL_FLUSH = 1The flush mode partial flush, 1.
SYNC_FLUSH
const SYNC_FLUSH = 2The flush mode sync flush, 2.
FULL_FLUSH
const FULL_FLUSH = 3The flush mode full flush, 3.
FINISH
const FINISH = 4The flush mode finish, 4.
BLOCK_FLUSH
const BLOCK_FLUSH = 5The flush mode block flush, 5.
CORRUPT
const CORRUPT = 1The input is not a valid stream.
TRUNCATED
const TRUNCATED = 2The input ends before the stream does.
TOO_LARGE
const TOO_LARGE = 3The output, or the window that a frame asks for, is over the limit of the caller.
CHECKSUM
const CHECKSUM = 4A checksum or a recorded length does not agree with the output.
UNSUPPORTED
const UNSUPPORTED = 5The stream uses a feature that the decoder does not have (a zlib preset dictionary).
NEEDS_DICTIONARY
const NEEDS_DICTIONARY = 6The frame needs a dictionary that the decoder does not have.
RAW
const RAW = 0DEFLATE framings: no framing (RFC 1951), zlib (RFC 1950) and gzip (RFC 1952).
ZLIB
const ZLIB = 1The zlib framing (RFC 1950).
GZIP
const GZIP = 2The gzip framing (RFC 1952).
ZSTD_WINDOW_LIMIT
const ZSTD_WINDOW_LIMIT = 134217728The default window limit, as the zstd tool's (2^27).
BROTLI_PROCESS
const BROTLI_PROCESS = 0BrotliEncoderOperation.
BROTLI_FLUSH
const BROTLI_FLUSH = 1The flush mode brotli flush, 1.
BROTLI_FINISH
const BROTLI_FINISH = 2The flush mode brotli finish, 2.
BROTLI_QUALITY
const BROTLI_QUALITY = 1BrotliEncoderParameter: the quality (0 to 11) and the window (log2 of its size, 10 to 24).
BROTLI_WINDOW
const BROTLI_WINDOW = 2The parameter of the window size, as log2 of its bytes (10 to 24).