# gizdb/sql/source

Files as tables: the one way GizDB reads data it does not store (DATABASE §6, sources). A file source is a glob over the engine's files directory (Engine.files) and a format: Parquet, CSV (and TSV, dbgen's '|' tables) or newline-delimited JSON. Opening it reads only what fixes the schema (Parquet footers, the first rows of a text file); a query then reads it in units (a Parquet row group, a text file) that the executor decodes in parallel, each into the same columns a stored segment holds, only the columns it reads, and skips Parquet row groups whose statistics rule out its conditions. SQL: SELECT * FROM 'sales/*.parquet', read_csv('a.csv', header => true).

```gizlang
import * as Source from "gizdb/sql/source";
```

## Types

### `TextOptions`

```gizlang
type TextOptions = { delimiter: u64; header: i64; quote: u64; }
```

Options of a text format: the delimiter (0: detect), whether the first line names the columns
(-1: detect), the quote.

- `delimiter`: `u64`
- `header`: `i64`
- `quote`: `u64`

### `Unit`

```gizlang
type Unit = { file: i64; group: i64; rows: i64; }
```

A part of a source read alone: a Parquet row group of a file, or a whole text file (`group` -1), and its rows.

- `file`: `i64`
- `group`: `i64`
- `rows`: `i64`

### `FileTable`

```gizlang
type FileTable = { format: i64; paths: Vec<string>; names: Vec<string>; types: Vec<Type>; text: TextOptions; metas: Vec<Pq.Meta>; units: Vec<Unit>; rows: i64; }
```

An opened source: its format, files, schema, text options, Parquet footers and units.

- `format`: `i64`
- `paths`: `Vec<string>`
- `names`: `Vec<string>`
- `types`: `Vec<M143.Type>`
- `text`: `M108.TextOptions`
- `metas`: `Vec<M149.Meta>`
- `units`: `Vec<M108.Unit>`
- `rows`: `i64` — Its rows: exact for Parquet, for text estimated from the files' sizes and the sample's lines.

### `FilePlanningLimits`

```gizlang
type FilePlanningLimits = { maxFiles: i64; maxPathBytes: i64; maxDepth: i64; maxEntries: i64; maxFooterBytes: i64; maxSampleBytes: i64; metadataBytes: i64; }
```

Planning retains bounded path/schema/footer metadata under the same issuer as
subsequent reads. The legacy 1 MiB schema sample is never silently shortened.
These grants are conservative logical admission, not measured allocator/RSS.

- `maxFiles`: `i64`
- `maxPathBytes`: `i64`
- `maxDepth`: `i64`
- `maxEntries`: `i64`
- `maxFooterBytes`: `i64`
- `maxSampleBytes`: `i64`
- `metadataBytes`: `i64`

### `PlannedFiles`

```gizlang
type PlannedFiles = { table: FileTable; lease: ResourceLease; closed: bool; }
```

A source opened by openFilesContext, with the lease that holds its metadata's credit until closeContext.

- `table`: `M108.FileTable`
- `lease`: `M87.ResourceLease`
- `closed`: `bool`

### `Range`

```gizlang
type Range = { col: i64; ilo: i64; ihi: i64; flo: f64; fhi: f64; floats: bool; }
```

The inclusive range a kept row's value of column `col` lies in: integers, or floats when `floats`.

- `col`: `i64`
- `ilo`: `i64`
- `ihi`: `i64`
- `flo`: `f64`
- `fhi`: `f64`
- `floats`: `bool`

### `Decoded`

```gizlang
type Decoded = { cols: Vec<Column>; n: i64; }
```

A unit's columns as decoded: those `need` marks (empty: all), the others empty, and its rows.

- `cols`: `Vec<M145.Column>`
- `n`: `i64`

### `PageColumn`

```gizlang
type PageColumn = { raw: bytes; at: i64; decoder: Pq.PageDecoder; pages: Vec<Column>; skip: i64; rows: i64; }
```

A column chunk of a Parquet row group, read whole and decoded a page at a time (exec.giz FileWaves): its bytes, the
next page's offset, its decoder, and its decoded rows not yet given: its pages, the first from row SKIP.

- `raw`: `bytes`
- `at`: `i64`
- `decoder`: `M149.PageDecoder`
- `pages`: `Vec<M145.Column>`
- `skip`: `i64`
- `rows`: `i64`

### `TextPart`

```gizlang
type TextPart = { data: bytes; next: i64; last: bool; }
```

A window of a text file: its records' bytes, where the next window starts, and whether it ends the file.

- `data`: `bytes`
- `next`: `i64`
- `last`: `bool`

### `FileCursorLimits`

```gizlang
type FileCursorLimits = { readBytes: i64; recordBytes: i64; pageBytes: i64; pageRows: i64; decodeBytes: i64; batchRows: i64; batchBytes: i64; }
```

External sources retain one incomplete text record or one decoded page per
selected Parquet column, never a whole file/row group. All credits are logical
conservative reservations; the runtime allocator remains the physical cap.

- `readBytes`: `i64`
- `recordBytes`: `i64`
- `pageBytes`: `i64`
- `pageRows`: `i64`
- `decodeBytes`: `i64`
- `batchRows`: `i64`
- `batchBytes`: `i64`

### `FileCursor`

```gizlang
type FileCursor = { disk: Disk; table: FileTable; need: Vec<bool>; limits: FileCursorLimits; lease: ResourceLease; file: Option<File>; unit: i64; offset: i64; size: i64; buffer: bytes; line: i64; header: bool; columns: Vec<FileColumnCursor>; groupRow: i64; rowAt: i64; index: Map<string, i64>; pending: Option<Vec<Value>>; done: bool; failed: bool; closed: bool; }
```

A read of a source's units in order: the open file, the text buffer or the Parquet column pages, and the lease of its
credit (fileCursorContext).

- `disk`: `M87.Disk`
- `table`: `M108.FileTable`
- `need`: `Vec<bool>`
- `limits`: `M108.FileCursorLimits`
- `lease`: `M87.ResourceLease`
- `file`: `_gizlang_sdk_core.Option<M87.File>`
- `unit`: `i64`
- `offset`: `i64`
- `size`: `i64`
- `buffer`: `bytes`
- `line`: `i64`
- `header`: `bool`
- `columns`: `Vec<M108.FileColumnCursor>`
- `groupRow`: `i64`
- `rowAt`: `i64`
- `index`: `_gizlang_sdk_collections.Map<string, i64>`
- `pending`: `_gizlang_sdk_core.Option<Vec<M143.Value>>`
- `done`: `bool`
- `failed`: `bool`
- `closed`: `bool`

### `FilePoll`

```gizlang
type FilePoll = { batch: Option<Decoded>; scanned: i64; done: bool; base: i64; unit: i64; }
```

What a poll gave: a batch or none, the work it did (at most its maxWork), whether the source ended, and the unit and
the position of the batch's first row.

- `batch`: `_gizlang_sdk_core.Option<M108.Decoded>`
- `scanned`: `i64`
- `done`: `bool`
- `base`: `i64`
- `unit`: `i64`

## Functions

### `textOptions`

```gizlang
function textOptions(): TextOptions
```

The default text options: the delimiter and the header detected, the quote a double quote.

Effects: fail.

### `emptyFiles`

```gizlang
function emptyFiles(): FileTable
```

A source of no files (before one is opened).

Effects: fail.

### `formatOf`

```gizlang
function formatOf(path: string): i64
```

The format a path names by its extension; 0 when it names none.

Effects: fail, alloc.

### `globFiles`

```gizlang
function globFiles(files: Dir, pattern: string): Vec<string> throws DbError
```

The files of `files` the glob `pattern` names ('*', '?', '[...]' within a path part, '**' any
number of directories), in path order. A name without a wildcard is that file.

Effects: fail, read, io, alloc.

### `openFiles`

```gizlang
function openFiles(files: Dir, pattern: string, format0: i64, text: TextOptions): FileTable throws DbError
```

Opens the source `pattern` names in format `format` (0: by the files' extension). Every file
must have the first one's columns.

Effects: fail, read, io, alloc.

### `openFilesContext`

```gizlang
function openFilesContext(files: Dir, pattern: string, format0: i64, text: TextOptions, limits: FilePlanningLimits, credits: ResourceContext): PlannedFiles throws DbError
```

The files of `files` that PATTERN names, opened as one source of format FORMAT0 (0: by extension) within LIMITS, its
metadata charged to CREDITS.

Effects: fail, read, write, console, io, alloc.

### `unitMayMatch`

```gizlang
function unitMayMatch(t: FileTable, u: Unit, ranges: Vec<Range>): bool
```

Whether unit `u` may hold a row the conditions keep: Parquet statistics outside one of RANGES rule the row group out.

Effects: fail.

### `unitBytes`

```gizlang
function unitBytes(t: FileTable, u: Unit, need: Vec<bool>): i64
```

The bytes of Parquet row group `u` that `readUnit` reads: its chunks of the columns `need` marks (empty: all).

Effects: fail.

### `pageColumns`

```gizlang
function pageColumns(files: Dir, t: FileTable, u: Unit, need: Vec<bool>): Vec<PageColumn> throws DbError
```

The chunks of row group `u`, those `need` marks (empty: all) read (readUnit), each to be decoded a page at a time.

Effects: fail, read, io, alloc.

### `decodePages`

```gizlang
function decodePages(t: FileTable, u: Unit, c: i64, p: PageColumn, rows: i64): PageColumn throws DbError
```

P, column C of row group `u`, with its pages decoded until it holds ROWS rows.

Effects: fail, alloc.

### `takeRows`

```gizlang
function takeRows(p: mut PageColumn, n: i64, ty: Type): Column
```

The first N rows P holds (N at least one), as one column of type TY; P keeps the rest. Each row is copied once: a page
stays until its last row is given, so a page of a whole row group is not copied again for each wave.

Effects: fail, alloc.

### `textPart`

```gizlang
function textPart(files: Dir, t: FileTable, u: Unit, offset: i64, window: i64): TextPart throws DbError
```

The window of text unit `u` from OFFSET, about WINDOW bytes, cut after its last record end; a record longer than the
window widens it. A file of any size is read a window at a time.

Effects: fail, read, io, alloc.

### `decodeText`

```gizlang
function decodeText(t: FileTable, u: Unit, data: bytes, need: Vec<bool>, cores: i64, first: bool): Decoded throws DbError
```

The records of a window of text unit `u` (textPart); FIRST: the window starts its file, whose header it holds. A CSV
window is parsed on CORES cores (csvColumns).

Effects: fail, alloc.

### `admitFileCursor`

```gizlang
function admitFileCursor(t: FileTable, need: Vec<bool>, limits: FileCursorLimits, available: i64): i64 throws DbError
```

Pure admission for the original file cursor; Graph planning and allocation
use this same geometry without creating a second budget or reader.

Effects: fail, alloc.

### `fileCursorContext`

```gizlang
function fileCursorContext(files: Dir, t: FileTable, need: Vec<bool>, limits: FileCursorLimits, credits: ResourceContext): FileCursor throws DbError
```

A cursor over T reading the columns NEED marks (empty: all) within LIMITS, its credit taken from CREDITS (admitFileCursor).

Effects: fail, read, write, console, io, alloc.

### `quotedCuts`

```gizlang
function quotedCuts(d: bytes, q: u64, parts: i64): Vec<i64>
```

Record ends that cut CSV D, which holds quotes, into about PARTS parts: [0], then each cut, ascending and below D's
length. A line end ends a record only outside quotes, and when the quote is its own escape (a doubled quote) the quote
state at a byte is the parity of the quotes before it (a quote inside an unquoted field opens quotes too, as
csvRecord reads it): every core counts the quotes of its part, then finds the first record end after its start.

Effects: fail, alloc.

### `fileSource`

```gizlang
function fileSource(directory: Dir, identity: string, pattern: string, format: i64, text: TextOptions, planning: FilePlanningLimits, limits: FileCursorLimits, context: ResourceContext): Source
```

A Source over the files PATTERN names: CSV, NDJSON and Parquet all use the bounded physical file reader.
The supplied context is the caller's original resource issuer. Mutable files
are not advertised as transaction snapshots.

Effects: fail, alloc.

## Methods

### `PlannedFiles.closeContext`

```gizlang
function PlannedFiles.closeContext(self: mut PlannedFiles, credits: ResourceContext): bool throws DbError
```

Releases the planned source's credit: true when it was open, false when it was closed already.

Effects: fail, read, write, console, io, alloc.

### `FileCursor.pollContext`

```gizlang
function FileCursor.pollContext(self: mut FileCursor, credits: ResourceContext, maxWork: i64, cancelled: bool): FilePoll throws DbError
```

Drop the previous returned batch before the next poll or close. cancelled is
the actor's canonical deadline/cancellation decision, checked before physical IO.

Effects: fail, read, write, io, alloc.

### `FileCursor.closeContext`

```gizlang
function FileCursor.closeContext(self: mut FileCursor, credits: ResourceContext): bool throws DbError
```

Closes the cursor's file and releases its credit: whether the release freed it, false when it was closed already.

Effects: fail, read, write, console, io, alloc.

## Constants

### `PARQUET`

```gizlang
const PARQUET: i64 = 1
```

The format of Parquet files (FileTable.format).

### `CSV`

```gizlang
const CSV: i64 = 2
```

The format of delimited text files: CSV, TSV and the like (FileTable.format).

### `NDJSON`

```gizlang
const NDJSON: i64 = 3
```

The format of newline-delimited JSON files (FileTable.format).

### `REMOTE`

```gizlang
const REMOTE: i64 = 4
```

Not a file: the rows of a foreign table, which the session takes from its server (exec.giz remoteWaves).

