GizStack
View .mdOpen in ChatGPTOpen in Claude

gizdb/sql/source

Files as tables: the one way GizDB reads data it does not store (DATABASE §6, sources). A file source is a glob over the engine's files directory (Engine.files) and a format: Parquet, CSV (and TSV, dbgen's '|' tables) or newline-delimited JSON. Opening it reads only what fixes the schema (Parquet footers, the first rows of a text file); a query then reads it in units (a Parquet row group, a text file) that the executor decodes in parallel, each into the same columns a stored segment holds, only the columns it reads, and skips Parquet row groups whose statistics rule out its conditions. SQL: SELECT * FROM 'sales/*.parquet', read_csv('a.csv', header => true).

gizlang
import * as Source from "gizdb/sql/source";

Types

TextOptions

gizlang
type TextOptions = { delimiter: u64; header: i64; quote: u64; }

Options of a text format: the delimiter (0: detect), whether the first line names the columns (-1: detect), the quote.

  • delimiter: u64
  • header: i64
  • quote: u64

Unit

gizlang
type Unit = { file: i64; group: i64; rows: i64; }

A part of a source read alone: a Parquet row group of a file, or a whole text file (group -1), and its rows.

  • file: i64
  • group: i64
  • rows: i64

FileTable

gizlang
type FileTable = { format: i64; paths: Vec<string>; names: Vec<string>; types: Vec<Type>; text: TextOptions; metas: Vec<Pq.Meta>; units: Vec<Unit>; rows: i64; }

An opened source: its format, files, schema, text options, Parquet footers and units.

  • format: i64
  • paths: Vec<string>
  • names: Vec<string>
  • types: Vec<M143.Type>
  • text: M108.TextOptions
  • metas: Vec<M149.Meta>
  • units: Vec<M108.Unit>
  • rows: i64 — Its rows: exact for Parquet, for text estimated from the files' sizes and the sample's lines.

FilePlanningLimits

gizlang
type FilePlanningLimits = { maxFiles: i64; maxPathBytes: i64; maxDepth: i64; maxEntries: i64; maxFooterBytes: i64; maxSampleBytes: i64; metadataBytes: i64; }

Planning retains bounded path/schema/footer metadata under the same issuer as subsequent reads. The legacy 1 MiB schema sample is never silently shortened. These grants are conservative logical admission, not measured allocator/RSS.

  • maxFiles: i64
  • maxPathBytes: i64
  • maxDepth: i64
  • maxEntries: i64
  • maxFooterBytes: i64
  • maxSampleBytes: i64
  • metadataBytes: i64

PlannedFiles

gizlang
type PlannedFiles = { table: FileTable; lease: ResourceLease; closed: bool; }

A source opened by openFilesContext, with the lease that holds its metadata's credit until closeContext.

  • table: M108.FileTable
  • lease: M87.ResourceLease
  • closed: bool

Range

gizlang
type Range = { col: i64; ilo: i64; ihi: i64; flo: f64; fhi: f64; floats: bool; }

The inclusive range a kept row's value of column col lies in: integers, or floats when floats.

  • col: i64
  • ilo: i64
  • ihi: i64
  • flo: f64
  • fhi: f64
  • floats: bool

Decoded

gizlang
type Decoded = { cols: Vec<Column>; n: i64; }

A unit's columns as decoded: those need marks (empty: all), the others empty, and its rows.

  • cols: Vec<M145.Column>
  • n: i64

PageColumn

gizlang
type PageColumn = { raw: bytes; at: i64; decoder: Pq.PageDecoder; pages: Vec<Column>; skip: i64; rows: i64; }

A column chunk of a Parquet row group, read whole and decoded a page at a time (exec.giz FileWaves): its bytes, the next page's offset, its decoder, and its decoded rows not yet given: its pages, the first from row SKIP.

  • raw: bytes
  • at: i64
  • decoder: M149.PageDecoder
  • pages: Vec<M145.Column>
  • skip: i64
  • rows: i64

TextPart

gizlang
type TextPart = { data: bytes; next: i64; last: bool; }

A window of a text file: its records' bytes, where the next window starts, and whether it ends the file.

  • data: bytes
  • next: i64
  • last: bool

FileCursorLimits

gizlang
type FileCursorLimits = { readBytes: i64; recordBytes: i64; pageBytes: i64; pageRows: i64; decodeBytes: i64; batchRows: i64; batchBytes: i64; }

External sources retain one incomplete text record or one decoded page per selected Parquet column, never a whole file/row group. All credits are logical conservative reservations; the runtime allocator remains the physical cap.

  • readBytes: i64
  • recordBytes: i64
  • pageBytes: i64
  • pageRows: i64
  • decodeBytes: i64
  • batchRows: i64
  • batchBytes: i64

FileCursor

gizlang
type FileCursor = { disk: Disk; table: FileTable; need: Vec<bool>; limits: FileCursorLimits; lease: ResourceLease; file: Option<File>; unit: i64; offset: i64; size: i64; buffer: bytes; line: i64; header: bool; columns: Vec<FileColumnCursor>; groupRow: i64; rowAt: i64; index: Map<string, i64>; pending: Option<Vec<Value>>; done: bool; failed: bool; closed: bool; }

A read of a source's units in order: the open file, the text buffer or the Parquet column pages, and the lease of its credit (fileCursorContext).

  • disk: M87.Disk
  • table: M108.FileTable
  • need: Vec<bool>
  • limits: M108.FileCursorLimits
  • lease: M87.ResourceLease
  • file: _gizlang_sdk_core.Option<M87.File>
  • unit: i64
  • offset: i64
  • size: i64
  • buffer: bytes
  • line: i64
  • header: bool
  • columns: Vec<M108.FileColumnCursor>
  • groupRow: i64
  • rowAt: i64
  • index: _gizlang_sdk_collections.Map<string, i64>
  • pending: _gizlang_sdk_core.Option<Vec<M143.Value>>
  • done: bool
  • failed: bool
  • closed: bool

FilePoll

gizlang
type FilePoll = { batch: Option<Decoded>; scanned: i64; done: bool; base: i64; unit: i64; }

What a poll gave: a batch or none, the work it did (at most its maxWork), whether the source ended, and the unit and the position of the batch's first row.

  • batch: _gizlang_sdk_core.Option<M108.Decoded>
  • scanned: i64
  • done: bool
  • base: i64
  • unit: i64

Functions

textOptions

gizlang
function textOptions(): TextOptions

The default text options: the delimiter and the header detected, the quote a double quote.

Effects: fail.

emptyFiles

gizlang
function emptyFiles(): FileTable

A source of no files (before one is opened).

Effects: fail.

formatOf

gizlang
function formatOf(path: string): i64

The format a path names by its extension; 0 when it names none.

Effects: fail, alloc.

globFiles

gizlang
function globFiles(files: Dir, pattern: string): Vec<string> throws DbError

The files of files the glob pattern names ('*', '?', '[...]' within a path part, '**' any number of directories), in path order. A name without a wildcard is that file.

Effects: fail, read, io, alloc.

openFiles

gizlang
function openFiles(files: Dir, pattern: string, format0: i64, text: TextOptions): FileTable throws DbError

Opens the source pattern names in format format (0: by the files' extension). Every file must have the first one's columns.

Effects: fail, read, io, alloc.

openFilesContext

gizlang
function openFilesContext(files: Dir, pattern: string, format0: i64, text: TextOptions, limits: FilePlanningLimits, credits: ResourceContext): PlannedFiles throws DbError

The files of files that PATTERN names, opened as one source of format FORMAT0 (0: by extension) within LIMITS, its metadata charged to CREDITS.

Effects: fail, read, write, console, io, alloc.

unitMayMatch

gizlang
function unitMayMatch(t: FileTable, u: Unit, ranges: Vec<Range>): bool

Whether unit u may hold a row the conditions keep: Parquet statistics outside one of RANGES rule the row group out.

Effects: fail.

unitBytes

gizlang
function unitBytes(t: FileTable, u: Unit, need: Vec<bool>): i64

The bytes of Parquet row group u that readUnit reads: its chunks of the columns need marks (empty: all).

Effects: fail.

pageColumns

gizlang
function pageColumns(files: Dir, t: FileTable, u: Unit, need: Vec<bool>): Vec<PageColumn> throws DbError

The chunks of row group u, those need marks (empty: all) read (readUnit), each to be decoded a page at a time.

Effects: fail, read, io, alloc.

decodePages

gizlang
function decodePages(t: FileTable, u: Unit, c: i64, p: PageColumn, rows: i64): PageColumn throws DbError

P, column C of row group u, with its pages decoded until it holds ROWS rows.

Effects: fail, alloc.

takeRows

gizlang
function takeRows(p: mut PageColumn, n: i64, ty: Type): Column

The first N rows P holds (N at least one), as one column of type TY; P keeps the rest. Each row is copied once: a page stays until its last row is given, so a page of a whole row group is not copied again for each wave.

Effects: fail, alloc.

textPart

gizlang
function textPart(files: Dir, t: FileTable, u: Unit, offset: i64, window: i64): TextPart throws DbError

The window of text unit u from OFFSET, about WINDOW bytes, cut after its last record end; a record longer than the window widens it. A file of any size is read a window at a time.

Effects: fail, read, io, alloc.

decodeText

gizlang
function decodeText(t: FileTable, u: Unit, data: bytes, need: Vec<bool>, cores: i64, first: bool): Decoded throws DbError

The records of a window of text unit u (textPart); FIRST: the window starts its file, whose header it holds. A CSV window is parsed on CORES cores (csvColumns).

Effects: fail, alloc.

admitFileCursor

gizlang
function admitFileCursor(t: FileTable, need: Vec<bool>, limits: FileCursorLimits, available: i64): i64 throws DbError

Pure admission for the original file cursor; Graph planning and allocation use this same geometry without creating a second budget or reader.

Effects: fail, alloc.

fileCursorContext

gizlang
function fileCursorContext(files: Dir, t: FileTable, need: Vec<bool>, limits: FileCursorLimits, credits: ResourceContext): FileCursor throws DbError

A cursor over T reading the columns NEED marks (empty: all) within LIMITS, its credit taken from CREDITS (admitFileCursor).

Effects: fail, read, write, console, io, alloc.

quotedCuts

gizlang
function quotedCuts(d: bytes, q: u64, parts: i64): Vec<i64>

Record ends that cut CSV D, which holds quotes, into about PARTS parts: [0], then each cut, ascending and below D's length. A line end ends a record only outside quotes, and when the quote is its own escape (a doubled quote) the quote state at a byte is the parity of the quotes before it (a quote inside an unquoted field opens quotes too, as csvRecord reads it): every core counts the quotes of its part, then finds the first record end after its start.

Effects: fail, alloc.

fileSource

gizlang
function fileSource(directory: Dir, identity: string, pattern: string, format: i64, text: TextOptions, planning: FilePlanningLimits, limits: FileCursorLimits, context: ResourceContext): Source

A Source over the files PATTERN names: CSV, NDJSON and Parquet all use the bounded physical file reader. The supplied context is the caller's original resource issuer. Mutable files are not advertised as transaction snapshots.

Effects: fail, alloc.

Methods

PlannedFiles.closeContext

gizlang
function PlannedFiles.closeContext(self: mut PlannedFiles, credits: ResourceContext): bool throws DbError

Releases the planned source's credit: true when it was open, false when it was closed already.

Effects: fail, read, write, console, io, alloc.

FileCursor.pollContext

gizlang
function FileCursor.pollContext(self: mut FileCursor, credits: ResourceContext, maxWork: i64, cancelled: bool): FilePoll throws DbError

Drop the previous returned batch before the next poll or close. cancelled is the actor's canonical deadline/cancellation decision, checked before physical IO.

Effects: fail, read, write, io, alloc.

FileCursor.closeContext

gizlang
function FileCursor.closeContext(self: mut FileCursor, credits: ResourceContext): bool throws DbError

Closes the cursor's file and releases its credit: whether the release freed it, false when it was closed already.

Effects: fail, read, write, console, io, alloc.

Constants

PARQUET

gizlang
const PARQUET: i64 = 1

The format of Parquet files (FileTable.format).

CSV

gizlang
const CSV: i64 = 2

The format of delimited text files: CSV, TSV and the like (FileTable.format).

NDJSON

gizlang
const NDJSON: i64 = 3

The format of newline-delimited JSON files (FileTable.format).

REMOTE

gizlang
const REMOTE: i64 = 4

Not a file: the rows of a foreign table, which the session takes from its server (exec.giz remoteWaves).