# gizdb:core/parquet Apache Parquet files read as GizDB columns (db/source.giz reads them as a table): the footer's schema and row groups, and each row group's column chunks decoded into column.giz columns, a dictionary-encoded text chunk keeping its dictionary as the column's. Flat schemas, and a list of floats or doubles as a vector; the codecs UNCOMPRESSED, SNAPPY, GZIP, ZSTD and LZ4_RAW, every encoding the format defines for them (PLAIN, the dictionary ones, RLE booleans, the three DELTA ones, BYTE_STREAM_SPLIT), data pages v1 and v2, and the logical types GizDB has: strings, integers, floats, booleans, decimals (up to 38 digits), dates, times and timestamps (MILLIS, MICROS, NANOS, INT96). Malformed input is an error, never a fault: every read is bounded by the bytes it has. The writer at the end of the file writes the same format back (see "the writer"). ```gizlang import * as Parquet from "gizdb:core/parquet"; ``` ## Types ### `T` ```gizlang type T = { k: i64; i: i64; d: f64; b: bytes; ids: Vec; kids: Vec; } ``` A decoded Thrift value: `k` its type (compact protocol ids: 1 true, 2 false, 3-6 integers, 7 double, 8 binary, 9 list, 10 set, 11 map, 12 struct); a struct holds field `ids` and `kids`. - `k`: `i64` - `i`: `i64` - `d`: `f64` - `b`: `bytes` - `ids`: `Vec` - `kids`: `Vec` ### `CompactLayout` ```gizlang type CompactLayout = { values: i64; fields: i64; binaryBytes: i64; depth: i64; } ``` Allocation geometry of the same compact grammar used by metadata/page readers. Measuring retains no decoded tree and does not tighten the reader's wire semantics. - `values`: `i64` - `fields`: `i64` - `binaryBytes`: `i64` - `depth`: `i64` ### `Field` ```gizlang type Field = { name: string; ty: Type; physical: i64; width: i64; optional: bool; scale: i64; unit: i64; list: i64; } ``` A column of the file: its name, the type GizDB reads it as, its physical type (0 BOOLEAN, 1 INT32, 2 INT64, 3 INT96, 4 FLOAT, 5 DOUBLE, 6 BYTE_ARRAY, 7 FIXED_LEN_BYTE_ARRAY) and width, whether it may hold NULLs, the decimal scale (-1: no decimal) and the time unit it is stored in (1 ms, 2 µs, 3 ns, 0 none; values are read as µs, days for a DATE). - `name`: `string` - `ty`: `M80.Type` - `physical`: `i64` - `width`: `i64` - `optional`: `bool` - `scale`: `i64` - `unit`: `i64` - `list`: `i64` ### `Chunk` ```gizlang type Chunk = { start: i64; size: i64; codec: i64; values: i64; min: bytes; max: bytes; nulls: i64; } ``` A column chunk of a row group: where its pages lie, their codec, value count and statistics (`min`/`max` plain-encoded, empty when the writer gave none). - `start`: `i64` - `size`: `i64` - `codec`: `i64` - `values`: `i64` - `min`: `bytes` - `max`: `bytes` - `nulls`: `i64` ### `RowGroup` ```gizlang type RowGroup = { rows: i64; chunks: Vec; } ``` A row group: its row count and one chunk per leaf column. - `rows`: `i64` - `chunks`: `Vec` ### `Meta` ```gizlang type Meta = { fields: Vec; rows: i64; groups: Vec; } ``` The file metadata: its leaf fields, its row count and its row groups. - `fields`: `Vec` - `rows`: `i64` - `groups`: `Vec` ### `PageHeader` ```gizlang type PageHeader = { value: T; bytes: i64; compressed: i64; plain: i64; kind: i64; rows: i64; encoding: i64; } ``` A bounded header parsed by the same Compact Thrift owner as resident chunks. The caller must admit the header window before parsing, then the body/decode credit. - `value`: `M90.T` - `bytes`: `i64` - `compressed`: `i64` - `plain`: `i64` - `kind`: `i64` - `rows`: `i64` - `encoding`: `i64` ### `PageDecoder` ```gizlang type PageDecoder = { dictionary: State; rows: i64; expected: i64; dictionaryBytes: i64; } ``` The dictionary alone survives page boundaries; plain data pages do not retain all earlier output dictionaries/rows. Source owns encoded offsets and page leases. - `dictionary`: `M90.State` - `rows`: `i64` - `expected`: `i64` - `dictionaryBytes`: `i64` ### `Writer` ```gizlang type Writer = { codec: i64; specs: Vec; rows: Vec>; held: i64; maxRows: i64; maxBytes: i64; offset: i64; started: bool; groups: Vec; total: i64; } ``` A Parquet file being written. `rows` are the rows of the open row group; `started` says that the leading magic is out; `groups` hold each finished RowGroup struct; `codec` compresses the pages (0 none, the default; 2 GZIP; 7 LZ4_RAW). - `codec`: `i64` - `specs`: `Vec` - `rows`: `Vec>` - `held`: `i64` - `maxRows`: `i64` - `maxBytes`: `i64` - `offset`: `i64` - `started`: `bool` - `groups`: `Vec` - `total`: `i64` ## Functions ### `compactLayout` ```gizlang function compactLayout(foot: bytes): CompactLayout throws string ``` Measure one footer struct using the canonical parser, before allocating its tree. Effects: fail. ### `footerLength` ```gizlang function footerLength(tail: bytes, size: i64): i64 throws string ``` The length of the footer of a file of `size` bytes whose last 8 bytes are `tail`. Effects: fail. ### `meta` ```gizlang function meta(foot: bytes): Meta throws string ``` The metadata in a file's footer (the `footerLength` bytes before its last 8). Effects: fail. ### `decodeChunk` ```gizlang function decodeChunk(f: Field, ch: Chunk, data: bytes, rows: i64): Column throws string ``` Column `f`'s chunk of `rows` rows, from its bytes (`data`, the chunk's `size` bytes from its start), as a column of `rows` values. Effects: fail. ### `pageHeader` ```gizlang function pageHeader(data: bytes, maxPageBytes: i64, maxPageRows: i64): PageHeader throws string ``` The page header at the start of DATA, refused when DATA is longer than MAXPAGEBYTES or the page has more than MAXPAGEROWS rows. Effects: fail. ### `pageDecoder` ```gizlang function pageDecoder(f: Field, rows: i64): PageDecoder throws string ``` A decoder of the pages of one chunk of F, which holds ROWS rows. Effects: fail. ### `pageDecodeCredit` ```gizlang function pageDecodeCredit(decoder: PageDecoder, h: PageHeader, allowance: i64): i64 throws string ``` Conservative logical overlap, including expanding delta prefixes and repeated binary dictionary values. This is admission credit, not allocator/RSS bytes. Effects: fail. ### `snappy` ```gizlang function snappy(data: bytes, size: i64): bytes throws string ``` Snappy's raw format: the uncompressed length, then literals and back-references. Effects: fail. ### `statInt` ```gizlang function statInt(f: Field, b: bytes): Option ``` A chunk's minimum or maximum as an integer on the column's own scale (µs for times, days for dates, coefficients for decimals), when its statistics give one in 8 bytes or fewer. Effects: fail. ### `statFloat` ```gizlang function statFloat(f: Field, b: bytes): Option ``` The statistic B of F as a float, when F is a float or double column. Effects: fail. ### `writer` ```gizlang function writer(names: Vec, types: Vec, maxRows: i64, maxBytes: i64): Writer throws string ``` A writer for columns `names` of `types`: a row group closes at `maxRows` rows or about `maxBytes` bytes of values, whichever comes first. Effects: fail. ## Methods ### `T.at` ```gizlang function T.at(self: T, id: i64): T ``` Field `id` of struct `t`, or an absent value (k 0). Effects: fail. ### `T.has` ```gizlang function T.has(self: T, id: i64): bool ``` Whether the struct has the field ID. Effects: fail. ### `PageDecoder.decode` ```gizlang function PageDecoder.decode(self: mut PageDecoder, ch: Chunk, h: PageHeader, body: bytes): Option throws string ``` The column of one page of the chunk CH with header H and body BODY; none for a dictionary or index page. Effects: fail. ### `PageDecoder.finish` ```gizlang function PageDecoder.finish(self: PageDecoder): bool throws string ``` True once the decoded pages hold as many rows as the chunk; an error otherwise. ### `Writer.add` ```gizlang function Writer.add(self: mut Writer, batch: Vec>): bytes throws string ``` Takes `batch` rows (one value for each column); returns the bytes of the file that are ready (the leading magic, then each row group that closed), possibly none. Effects: fail. ### `Writer.finish` ```gizlang function Writer.finish(self: mut Writer): bytes throws string ``` Closes the last row group and returns the rest of the file: the footer and the trailing magic. Effects: fail. ## Constants ### `COMPACT_MAX_DEPTH` ```gizlang const COMPACT_MAX_DEPTH = 64 ``` The deepest nesting of compact-protocol structs and lists that a reader accepts.