gizdb/sql/source
Files as tables: the one way GizDB reads data it does not store (DATABASE §6, sources). A file source is a glob over the engine's files directory (Engine.files) and a format: Parquet, CSV (and TSV, dbgen's '|' tables) or newline-delimited JSON. Opening it reads only what fixes the schema (Parquet footers, the first rows of a text file); a query then reads it in units (a Parquet row group, a text file) that the executor decodes in parallel, each into the same columns a stored segment holds, only the columns it reads, and skips Parquet row groups whose statistics rule out its conditions. SQL: SELECT * FROM 'sales/*.parquet', read_csv('a.csv', header => true).
import * as Source from "gizdb/sql/source";Types
TextOptions
type TextOptions = { delimiter: u64; header: i64; quote: u64; }Options of a text format: the delimiter (0: detect), whether the first line names the columns (-1: detect), the quote.
delimiter:u64header:i64quote:u64
Unit
type Unit = { file: i64; group: i64; rows: i64; }A part of a source read alone: a Parquet row group of a file, or a whole text file (group -1), and its rows.
file:i64group:i64rows:i64
FileTable
type FileTable = { format: i64; paths: Vec<string>; names: Vec<string>; types: Vec<Type>; text: TextOptions; metas: Vec<Pq.Meta>; units: Vec<Unit>; rows: i64; }An opened source: its format, files, schema, text options, Parquet footers and units.
format:i64paths:Vec<string>names:Vec<string>types:Vec<M143.Type>text:M108.TextOptionsmetas:Vec<M149.Meta>units:Vec<M108.Unit>rows:i64— Its rows: exact for Parquet, for text estimated from the files' sizes and the sample's lines.
FilePlanningLimits
type FilePlanningLimits = { maxFiles: i64; maxPathBytes: i64; maxDepth: i64; maxEntries: i64; maxFooterBytes: i64; maxSampleBytes: i64; metadataBytes: i64; }Planning retains bounded path/schema/footer metadata under the same issuer as subsequent reads. The legacy 1 MiB schema sample is never silently shortened. These grants are conservative logical admission, not measured allocator/RSS.
maxFiles:i64maxPathBytes:i64maxDepth:i64maxEntries:i64maxFooterBytes:i64maxSampleBytes:i64metadataBytes:i64
PlannedFiles
type PlannedFiles = { table: FileTable; lease: ResourceLease; closed: bool; }A source opened by openFilesContext, with the lease that holds its metadata's credit until closeContext.
table:M108.FileTablelease:M87.ResourceLeaseclosed:bool
Range
type Range = { col: i64; ilo: i64; ihi: i64; flo: f64; fhi: f64; floats: bool; }The inclusive range a kept row's value of column col lies in: integers, or floats when floats.
col:i64ilo:i64ihi:i64flo:f64fhi:f64floats:bool
Decoded
type Decoded = { cols: Vec<Column>; n: i64; }A unit's columns as decoded: those need marks (empty: all), the others empty, and its rows.
cols:Vec<M145.Column>n:i64
PageColumn
type PageColumn = { raw: bytes; at: i64; decoder: Pq.PageDecoder; pages: Vec<Column>; skip: i64; rows: i64; }A column chunk of a Parquet row group, read whole and decoded a page at a time (exec.giz FileWaves): its bytes, the next page's offset, its decoder, and its decoded rows not yet given: its pages, the first from row SKIP.
raw:bytesat:i64decoder:M149.PageDecoderpages:Vec<M145.Column>skip:i64rows:i64
TextPart
type TextPart = { data: bytes; next: i64; last: bool; }A window of a text file: its records' bytes, where the next window starts, and whether it ends the file.
data:bytesnext:i64last:bool
FileCursorLimits
type FileCursorLimits = { readBytes: i64; recordBytes: i64; pageBytes: i64; pageRows: i64; decodeBytes: i64; batchRows: i64; batchBytes: i64; }External sources retain one incomplete text record or one decoded page per selected Parquet column, never a whole file/row group. All credits are logical conservative reservations; the runtime allocator remains the physical cap.
readBytes:i64recordBytes:i64pageBytes:i64pageRows:i64decodeBytes:i64batchRows:i64batchBytes:i64
FileCursor
type FileCursor = { disk: Disk; table: FileTable; need: Vec<bool>; limits: FileCursorLimits; lease: ResourceLease; file: Option<File>; unit: i64; offset: i64; size: i64; buffer: bytes; line: i64; header: bool; columns: Vec<FileColumnCursor>; groupRow: i64; rowAt: i64; index: Map<string, i64>; pending: Option<Vec<Value>>; done: bool; failed: bool; closed: bool; }A read of a source's units in order: the open file, the text buffer or the Parquet column pages, and the lease of its credit (fileCursorContext).
disk:M87.Disktable:M108.FileTableneed:Vec<bool>limits:M108.FileCursorLimitslease:M87.ResourceLeasefile:_gizlang_sdk_core.Option<M87.File>unit:i64offset:i64size:i64buffer:bytesline:i64header:boolcolumns:Vec<M108.FileColumnCursor>groupRow:i64rowAt:i64index:_gizlang_sdk_collections.Map<string, i64>pending:_gizlang_sdk_core.Option<Vec<M143.Value>>done:boolfailed:boolclosed:bool
FilePoll
type FilePoll = { batch: Option<Decoded>; scanned: i64; done: bool; base: i64; unit: i64; }What a poll gave: a batch or none, the work it did (at most its maxWork), whether the source ended, and the unit and the position of the batch's first row.
batch:_gizlang_sdk_core.Option<M108.Decoded>scanned:i64done:boolbase:i64unit:i64
Functions
textOptions
function textOptions(): TextOptionsThe default text options: the delimiter and the header detected, the quote a double quote.
Effects: fail.
emptyFiles
function emptyFiles(): FileTableA source of no files (before one is opened).
Effects: fail.
formatOf
function formatOf(path: string): i64The format a path names by its extension; 0 when it names none.
Effects: fail, alloc.
globFiles
function globFiles(files: Dir, pattern: string): Vec<string> throws DbErrorThe files of files the glob pattern names ('*', '?', '[...]' within a path part, '**' any
number of directories), in path order. A name without a wildcard is that file.
Effects: fail, read, io, alloc.
openFiles
function openFiles(files: Dir, pattern: string, format0: i64, text: TextOptions): FileTable throws DbErrorOpens the source pattern names in format format (0: by the files' extension). Every file
must have the first one's columns.
Effects: fail, read, io, alloc.
openFilesContext
function openFilesContext(files: Dir, pattern: string, format0: i64, text: TextOptions, limits: FilePlanningLimits, credits: ResourceContext): PlannedFiles throws DbErrorThe files of files that PATTERN names, opened as one source of format FORMAT0 (0: by extension) within LIMITS, its
metadata charged to CREDITS.
Effects: fail, read, write, console, io, alloc.
unitMayMatch
function unitMayMatch(t: FileTable, u: Unit, ranges: Vec<Range>): boolWhether unit u may hold a row the conditions keep: Parquet statistics outside one of RANGES rule the row group out.
Effects: fail.
unitBytes
function unitBytes(t: FileTable, u: Unit, need: Vec<bool>): i64The bytes of Parquet row group u that readUnit reads: its chunks of the columns need marks (empty: all).
Effects: fail.
pageColumns
function pageColumns(files: Dir, t: FileTable, u: Unit, need: Vec<bool>): Vec<PageColumn> throws DbErrorThe chunks of row group u, those need marks (empty: all) read (readUnit), each to be decoded a page at a time.
Effects: fail, read, io, alloc.
decodePages
function decodePages(t: FileTable, u: Unit, c: i64, p: PageColumn, rows: i64): PageColumn throws DbErrorP, column C of row group u, with its pages decoded until it holds ROWS rows.
Effects: fail, alloc.
takeRows
function takeRows(p: mut PageColumn, n: i64, ty: Type): ColumnThe first N rows P holds (N at least one), as one column of type TY; P keeps the rest. Each row is copied once: a page stays until its last row is given, so a page of a whole row group is not copied again for each wave.
Effects: fail, alloc.
textPart
function textPart(files: Dir, t: FileTable, u: Unit, offset: i64, window: i64): TextPart throws DbErrorThe window of text unit u from OFFSET, about WINDOW bytes, cut after its last record end; a record longer than the
window widens it. A file of any size is read a window at a time.
Effects: fail, read, io, alloc.
decodeText
function decodeText(t: FileTable, u: Unit, data: bytes, need: Vec<bool>, cores: i64, first: bool): Decoded throws DbErrorThe records of a window of text unit u (textPart); FIRST: the window starts its file, whose header it holds. A CSV
window is parsed on CORES cores (csvColumns).
Effects: fail, alloc.
admitFileCursor
function admitFileCursor(t: FileTable, need: Vec<bool>, limits: FileCursorLimits, available: i64): i64 throws DbErrorPure admission for the original file cursor; Graph planning and allocation use this same geometry without creating a second budget or reader.
Effects: fail, alloc.
fileCursorContext
function fileCursorContext(files: Dir, t: FileTable, need: Vec<bool>, limits: FileCursorLimits, credits: ResourceContext): FileCursor throws DbErrorA cursor over T reading the columns NEED marks (empty: all) within LIMITS, its credit taken from CREDITS (admitFileCursor).
Effects: fail, read, write, console, io, alloc.
quotedCuts
function quotedCuts(d: bytes, q: u64, parts: i64): Vec<i64>Record ends that cut CSV D, which holds quotes, into about PARTS parts: [0], then each cut, ascending and below D's length. A line end ends a record only outside quotes, and when the quote is its own escape (a doubled quote) the quote state at a byte is the parity of the quotes before it (a quote inside an unquoted field opens quotes too, as csvRecord reads it): every core counts the quotes of its part, then finds the first record end after its start.
Effects: fail, alloc.
fileSource
function fileSource(directory: Dir, identity: string, pattern: string, format: i64, text: TextOptions, planning: FilePlanningLimits, limits: FileCursorLimits, context: ResourceContext): SourceA Source over the files PATTERN names: CSV, NDJSON and Parquet all use the bounded physical file reader. The supplied context is the caller's original resource issuer. Mutable files are not advertised as transaction snapshots.
Effects: fail, alloc.
Methods
PlannedFiles.closeContext
function PlannedFiles.closeContext(self: mut PlannedFiles, credits: ResourceContext): bool throws DbErrorReleases the planned source's credit: true when it was open, false when it was closed already.
Effects: fail, read, write, console, io, alloc.
FileCursor.pollContext
function FileCursor.pollContext(self: mut FileCursor, credits: ResourceContext, maxWork: i64, cancelled: bool): FilePoll throws DbErrorDrop the previous returned batch before the next poll or close. cancelled is the actor's canonical deadline/cancellation decision, checked before physical IO.
Effects: fail, read, write, io, alloc.
FileCursor.closeContext
function FileCursor.closeContext(self: mut FileCursor, credits: ResourceContext): bool throws DbErrorCloses the cursor's file and releases its credit: whether the release freed it, false when it was closed already.
Effects: fail, read, write, console, io, alloc.
Constants
PARQUET
const PARQUET: i64 = 1The format of Parquet files (FileTable.format).
CSV
const CSV: i64 = 2The format of delimited text files: CSV, TSV and the like (FileTable.format).
NDJSON
const NDJSON: i64 = 3The format of newline-delimited JSON files (FileTable.format).
REMOTE
const REMOTE: i64 = 4Not a file: the rows of a foreign table, which the session takes from its server (exec.giz remoteWaves).