gizdb:core/parquet

Apache Parquet files read as GizDB columns (db/source.giz reads them as a table): the footer's schema and row groups, and each row group's column chunks decoded into column.giz columns, a dictionary-encoded text chunk keeping its dictionary as the column's. Flat schemas, and a list of floats or doubles as a vector; the codecs UNCOMPRESSED, SNAPPY, GZIP, ZSTD and LZ4_RAW, every encoding the format defines for them (PLAIN, the dictionary ones, RLE booleans, the three DELTA ones, BYTE_STREAM_SPLIT), data pages v1 and v2, and the logical types GizDB has: strings, integers, floats, booleans, decimals (up to 38 digits), dates, times and timestamps (MILLIS, MICROS, NANOS, INT96). Malformed input is an error, never a fault: every read is bounded by the bytes it has. The writer at the end of the file writes the same format back (see "the writer").

import * as Parquet from "gizdb:core/parquet";

Types

T

type T = { k: i64; i: i64; d: f64; b: bytes; ids: Vec<i64>; kids: Vec<T>; }

A decoded Thrift value: k its type (compact protocol ids: 1 true, 2 false, 3-6 integers, 7 double, 8 binary, 9 list, 10 set, 11 map, 12 struct); a struct holds field ids and kids.

• k: i64
• i: i64
• d: f64
• b: bytes
• ids: Vec<i64>
• kids: Vec<M90.T>

CompactLayout

type CompactLayout = { values: i64; fields: i64; binaryBytes: i64; depth: i64; }

Allocation geometry of the same compact grammar used by metadata/page readers. Measuring retains no decoded tree and does not tighten the reader's wire semantics.

• values: i64
• fields: i64
• binaryBytes: i64
• depth: i64

Field

type Field = { name: string; ty: Type; physical: i64; width: i64; optional: bool; scale: i64; unit: i64; list: i64; }

A column of the file: its name, the type GizDB reads it as, its physical type (0 BOOLEAN, 1 INT32, 2 INT64, 3 INT96, 4 FLOAT, 5 DOUBLE, 6 BYTE_ARRAY, 7 FIXED_LEN_BYTE_ARRAY) and width, whether it may hold NULLs, the decimal scale (-1: no decimal) and the time unit it is stored in (1 ms, 2 µs, 3 ns, 0 none; values are read as µs, days for a DATE).

• name: string
• ty: M80.Type
• physical: i64
• width: i64
• optional: bool
• scale: i64
• unit: i64
• list: i64

Chunk

type Chunk = { start: i64; size: i64; codec: i64; values: i64; min: bytes; max: bytes; nulls: i64; }

A column chunk of a row group: where its pages lie, their codec, value count and statistics (min/max plain-encoded, empty when the writer gave none).

• start: i64
• size: i64
• codec: i64
• values: i64
• min: bytes
• max: bytes
• nulls: i64

RowGroup

type RowGroup = { rows: i64; chunks: Vec<Chunk>; }

A row group: its row count and one chunk per leaf column.

• rows: i64
• chunks: Vec<M90.Chunk>

Meta

type Meta = { fields: Vec<Field>; rows: i64; groups: Vec<RowGroup>; }

The file metadata: its leaf fields, its row count and its row groups.

• fields: Vec<M90.Field>
• rows: i64
• groups: Vec<M90.RowGroup>

PageHeader

type PageHeader = { value: T; bytes: i64; compressed: i64; plain: i64; kind: i64; rows: i64; encoding: i64; }

A bounded header parsed by the same Compact Thrift owner as resident chunks. The caller must admit the header window before parsing, then the body/decode credit.

• value: M90.T
• bytes: i64
• compressed: i64
• plain: i64
• kind: i64
• rows: i64
• encoding: i64

PageDecoder

type PageDecoder = { dictionary: State; rows: i64; expected: i64; dictionaryBytes: i64; }

The dictionary alone survives page boundaries; plain data pages do not retain all earlier output dictionaries/rows. Source owns encoded offsets and page leases.

• dictionary: M90.State
• rows: i64
• expected: i64
• dictionaryBytes: i64

Writer

type Writer = { codec: i64; specs: Vec<Spec>; rows: Vec<Vec<Value>>; held: i64; maxRows: i64; maxBytes: i64; offset: i64; started: bool; groups: Vec<bytes>; total: i64; }

A Parquet file being written. rows are the rows of the open row group; started says that the leading magic is out; groups hold each finished RowGroup struct; codec compresses the pages (0 none, the default; 2 GZIP; 7 LZ4_RAW).

• codec: i64
• specs: Vec<M90.Spec>
• rows: Vec<Vec<M80.Value>>
• held: i64
• maxRows: i64
• maxBytes: i64
• offset: i64
• started: bool
• groups: Vec<bytes>
• total: i64

Functions

compactLayout

function compactLayout(foot: bytes): CompactLayout throws string

Measure one footer struct using the canonical parser, before allocating its tree.

Effects: fail.

footerLength

function footerLength(tail: bytes, size: i64): i64 throws string

The length of the footer of a file of size bytes whose last 8 bytes are tail.

Effects: fail.

meta

function meta(foot: bytes): Meta throws string

The metadata in a file's footer (the footerLength bytes before its last 8).

Effects: fail.

decodeChunk

function decodeChunk(f: Field, ch: Chunk, data: bytes, rows: i64): Column throws string

Column f's chunk of rows rows, from its bytes (data, the chunk's size bytes from its start), as a column of rows values.

Effects: fail.

pageHeader

function pageHeader(data: bytes, maxPageBytes: i64, maxPageRows: i64): PageHeader throws string

The page header at the start of DATA, refused when DATA is longer than MAXPAGEBYTES or the page has more than MAXPAGEROWS rows.

Effects: fail.

pageDecoder

function pageDecoder(f: Field, rows: i64): PageDecoder throws string

A decoder of the pages of one chunk of F, which holds ROWS rows.

Effects: fail.

pageDecodeCredit

function pageDecodeCredit(decoder: PageDecoder, h: PageHeader, allowance: i64): i64 throws string

Conservative logical overlap, including expanding delta prefixes and repeated binary dictionary values. This is admission credit, not allocator/RSS bytes.

Effects: fail.

snappy

function snappy(data: bytes, size: i64): bytes throws string

Snappy's raw format: the uncompressed length, then literals and back-references.

Effects: fail.

statInt

function statInt(f: Field, b: bytes): Option<i64>

A chunk's minimum or maximum as an integer on the column's own scale (µs for times, days for dates, coefficients for decimals), when its statistics give one in 8 bytes or fewer.

Effects: fail.

statFloat

function statFloat(f: Field, b: bytes): Option<f64>

The statistic B of F as a float, when F is a float or double column.

Effects: fail.

writer

function writer(names: Vec<string>, types: Vec<Type>, maxRows: i64, maxBytes: i64): Writer throws string

A writer for columns names of types: a row group closes at maxRows rows or about maxBytes bytes of values, whichever comes first.

Effects: fail.

Methods

T.at

function T.at(self: T, id: i64): T

Field id of struct t, or an absent value (k 0).

Effects: fail.

T.has

function T.has(self: T, id: i64): bool

Whether the struct has the field ID.

Effects: fail.

PageDecoder.decode

function PageDecoder.decode(self: mut PageDecoder, ch: Chunk, h: PageHeader, body: bytes): Option<Column> throws string

The column of one page of the chunk CH with header H and body BODY; none for a dictionary or index page.

Effects: fail.

PageDecoder.finish

function PageDecoder.finish(self: PageDecoder): bool throws string

True once the decoded pages hold as many rows as the chunk; an error otherwise.

Writer.add

function Writer.add(self: mut Writer, batch: Vec<Vec<Value>>): bytes throws string

Takes batch rows (one value for each column); returns the bytes of the file that are ready (the leading magic, then each row group that closed), possibly none.

Effects: fail.

Writer.finish

function Writer.finish(self: mut Writer): bytes throws string

Closes the last row group and returns the rest of the file: the footer and the trailing magic.

Effects: fail.

Constants

COMPACT_MAX_DEPTH

const COMPACT_MAX_DEPTH = 64

The deepest nesting of compact-protocol structs and lists that a reader accepts.