# GizDB: one database for records, analytics and search

GizDB gives transactions, analytics, full-text search, vector search, and live data from other systems, in one engine
written in Gizlang. It runs embedded in a process, or as a server that speaks the PostgreSQL, Redis and Kafka
protocols, with the same format.

This page is the design as it is built. For each decision, it gives the reason and what the decision replaces.

- A part is described only when its code and a committed test exist.
- A performance statement occurs only with a measurement in `bench/db/` (refer to the README there).
- The code is in `db/`. Its tests are the `*.test.giz` files and the suites in `tests/db` (`bin/giz test db`); each scenario of a fixture such as `pg.giz` is a test in `pg.test.giz`.

## Contents

0. [From the requirement and the physics](#0-from-the-requirement-and-the-physics)
1. [Seven primitives, one owner each](#1-seven-primitives-one-owner-each)
2. [Log](#2-log)
3. [Version and transactions](#3-version-and-transactions)
4. [Segment: one storage format](#4-segment-one-storage-format)
5. [Derived structures and watermarks](#5-derived-structures-and-watermarks)
6. [Plan: one algebra](#6-plan-one-algebra)
7. [Sources](#7-sources)
8. [Correctness](#8-correctness)
9. [Text analysis](#9-text-analysis)
10. [What GizDB does not do](#10-what-gizdb-does-not-do)
11. [Native versioned journal](#11-native-versioned-journal)
12. [Native storage resources](#12-native-storage-resources)

## Status

| Part | State |
| --- | --- |
| Values, types, schemas and their binary encoding (`db/value.giz`) | Built |
| Columns and column builders (`db/core/column.giz`) | Built |
| Text analysis v1 (`db/core/analyzer.giz`) | Built |
| Inverted index and Block-Max WAND top-k (`db/core/inverted.giz`) | Built. Equal to exhaustive scoring, ties included, and independent of segment splits. |
| Log (`db/core/log.giz`) | Built. Tested with a cut at every byte, every bit flipped, and a zero tail. |
| Shared binary encoding: checked frames, order-preserving keys (`fs/codec.giz`) | Built |
| Segments, delta, key index, transactions, checkpoint (`db/core/segment.giz`, `db/core/table.giz`, `db/core/engine.giz`) | Built. 3000 random commits with syncs, checkpoints, merges and reopens are equal to a model. Tested with a snapshot held across checkpoints, conflicts, and recovery from a log cut at every byte. |
| Ordered index: a B+tree of order-preserving keys with subtree counts (`fs/ordered.giz`) | Built. Walks, ranks and seeks are equal to a sorted model over 20000 random puts and removes. |
| Btree secondary indexes and range walks (`db/core/table.giz`, ranges in `db/sql/exec/optimize.giz`) | Built. Ranges, fractional and descending bounds, and ORDER BY ... LIMIT are equal to SQLite (`tests/db/sql.giz`). |
| Exact DECIMAL, NUMERIC and MONEY (`db/decimal.giz`) | Built. A maximum of 40 significant digits and scale 130 (the digits of Oracle NUMBER and its smallest exponent; its integers past 10^40 are not held). Past them, an error (22003), never a silent float. The Oracle dialect rounds each result as NUMBER does: 20 base-100 digits from the first nonzero one (`numberArith`). |
| Redis-protocol server (`db/kv`, §1) | Built. The replies of every data type and snapshot restore are in `tests/db`. Rates against Redis and Valkey are in `bench/db/kv`. |
| SQL, plan, operators (`db/sql`) | Built. Shared queries agree with SQLite (`tests/db/sql.giz`); the limits are in §6. |
| Vector index (`db/core/vector.giz`) | Built. Guide: [vector](../guides/vector.md).  |

## 0. From the requirement and the physics

A user needs these things:

- A write is durable, and visible to every later read.
- A read sees one consistent state.
- A question about the data gets an answer at the cost of the data that it must touch. For
  example: a record by key, a sum over a billion rows, or the documents that best agree with
  words or an image.
- Data from other systems is current, without a pipeline that someone must build.
- The loss of a disk, a machine or a region loses no committed write.
- The same model serves an embedded app, a server and a cluster.

These limits decide the design:

| Limit | Value | Result |
| --- | --- | --- |
| Light between regions | 50 to 150 ms round trip | A strongly consistent commit costs one round trip to a quorum. Nothing is faster. Placement puts the quorum near the writer. |
| NVMe | Approximately 10 µs, approximately 1M IO/s, sequential GB/s | Write sequentially (a log). Read immutable files. Sync in batches (group commit). |
| DRAM, caches | Approximately 100 ns, approximately 1 ns | Columns, compressed and processed in batches. Positions in place of copies. |
| fsync | 0.1 to 2 ms | Many transactions share one sync. |
| A coherent shared cache over a network (Cache Fusion of Oracle RAC) | One round trip for each contended block | Not used. Shared nothing: each range has one leader. |

## 1. Seven primitives, one owner each

Every product above implements the same few things separately. GizDB implements each one time.
All other features are combinations.

| Primitive | The one mechanism | Owns | Replaces |
| --- | --- | --- | --- |
| **Log** | An append-only, checksummed sequence of commits for each range. The log is the database (Aurora, Neon). | Durability, recovery, replication, change feeds, point-in-time restore | WAL, replication stream, CDC tool and backup format as separate things |
| **Version** | Every row version has `[begin, end)` commit timestamps (Hyper, Umbra). A snapshot is a timestamp. | Isolation, time travel, branches | Undo segments, and vacuum as a separate process |
| **Segment** | One immutable columnar file format, plus one row delta in memory. Compaction folds the delta into segments. | Point reads and scans from one copy | A row store next to a column store (TiKV next to TiFlash), ETL into a warehouse |
| **Derived** | A function from a segment to a structure, built with it and compacted with it: primary key index, zone maps, inverted index, vector graph, materialized view. Each one records its **watermark**. | Every index, search and precomputed result | A search cluster (Elasticsearch) or vector database that is synchronized by hand |
| **Plan** | One algebra over column batches: scan, filter, project, join, aggregate, window, **score**, **top-k** and **subscribe**. SQL and typed Gizlang queries are its two front ends. | OLTP lookups, analytics, hybrid search, live queries | A search DSL, a vector API, a streaming SQL dialect |
| **Source** | A table whose rows are in a different system. It is scanned remotely with pushdown, or followed through its change log into the local Log. | Federation, CDC, files, lakes | A connector pipeline for each system |
| **Policy** | Tenant, row and column rules, compiled into every plan as predicates before an index is probed | Permissions, isolation, residency | Filters applied to results after the query, ACL code for each feature |

### Where the code is

The files at the top of `db/` are the public boundary (`PACKAGES`: access, data, query, typed, value and the
others). The rest is in folders: `db/core/` (log, version, segment, tables, catalog, backup), `db/server/` (the
server, its actors and metrics), `db/sql/` (lex, parse, bind, sessions and statements) with `db/sql/types/` (SQL
types and their values), `db/sql/functions/` (built-in functions and aggregates), `db/sql/exec/` (plan,
optimizer and executor) and `db/sql/catalog/` (system catalogs and catalog objects), and one folder for each
protocol or source (`db/pg/`, `db/kv/`, `db/kafka/`, `db/ora/`, `db/ingest/`, `db/ml/`).

### One engine, several interfaces

SQL (the PostgreSQL wire and the Gizlang API) and the Redis protocol are front ends of the same
engine. They are not two databases.

- The Redis key space is the relations `_giz_kv` (one row a key), `_giz_kv_items`, `_giz_kv_ordered`, `_giz_kv_ttl`
  and `_giz_kv_size` ([db/kv/README.md](../../db/kv/README.md#data-layout)). SQL reads and writes the same rows, in the
  same transactions and snapshots.
- One copy of the data, one log contract and one recovery. The physical paths differ by work: a SET is a key mutation
  on a hash primary, a shipped autocommit UPDATE a key mutation on an ordered primary, and other SQL uses the ordered
  and columnar paths.
- With KV shards, each shard engine writes its keys alone, and the main engine reads and writes all shards' rows
  through parent relations in coordinated commits ([db/kv/README.md](../../db/kv/README.md#shards)).

Three engine mechanisms give key-value speed. OLTP needs each of them also:

1. The newest version of a row is updated in place when no open snapshot is older than it
   (HyPer). An older version is kept only for a snapshot that needs it. Thus a hot counter does
   not collect versions.
2. A container change is logged as its operation (`HSET k f v`) and replayed
   deterministically. It is not logged as the full value.
3. An ordered index serves ranges.

### The rule of orthogonality

A new capability adds an operator to Plan or a function to Derived. It never adds a storage
format, a log, a transaction mechanism or a permission path. A text search, a vector search and
a sum are three plans over the same versions.

## 2. Log

**Built**: `db/core/log.giz`.

### Format

- A position in the log is an **LSN**: the byte offset of a record in the full history of the
  log. Thus it only increases.
- The log is a series of files, each named by its first LSN.
- A record is `[length u32][crc u32][body]`. The crc is the CRC-32 (`gizlang:compress`) of the
  length bytes and the body, masked as LevelDB masks it. Thus a tail filled with zeros, or an
  embedded frame, never verifies. Length 0 is never written.
- A body is one commit: its timestamp and its row operations for each table. Its maximum is
  4 GiB (the range of the length). A larger commit fails with 54000.
- The operations of a commit are kept encoded as the log writes them, and framed where they
  are. The checksum of the parts is joined arithmetically. Thus the log buffer holds no second
  copy of a large commit.

### Recovery

Recovery reads records from the LSN of the checkpoint.

- The log file grows by 4 MiB of written zeros at a time (`File.writeAhead`). An append then
  overwrites zeros, and its sync flushes only data: a 4 KiB synced write on backup takes 1.8 ms
  into written zeros and 6.2 ms when it extends the file. A zero rest of a file is where its
  records end.
- A record that fails is a torn tail only when no valid record follows it in the last file. It
  is then cut off.
- A failed record with valid records after it, or a gap between files, is corruption. It stops
  the open. A committed record is never dropped silently.

The candidate recovery API returns metadata (`Opened.records/from`, a cursor and `FrameRange`),
not a vector of bodies.

- CRC verification, the header search in a damaged tail, and the canonical file-backed Reader
  windows read a maximum of 1 MiB for each physical read.
- Short reads continue. An early end of file, compared with the sampled file size, is an I/O
  error. It cannot authorize a truncation of the tail.
- The bytewise search keeps its header window. CRC candidates use independent bounded ranges.
- The Engine validates a complete commit and its Journal offset pairs in sequence. Then it
  applies them through the same mutation owner that live receive uses.
- A writable reopen still syncs before it publishes durability. An inspection never repairs or syncs.

> **Note:** The allocations of Values and schemas for each operation, and the resident table and index state, are
> separate resource work. These windows do not prove fixed total RAM or a capacity of terabytes.

### Commits that answer before their sync

By default a commit answers after the sync of its log record. `gizdb --synchronous-commit off`
answers once the record is in the log file, before its sync, as PostgreSQL's
`synchronous_commit = off` does (TechEmpower configures PostgreSQL so). A crash of gizdb loses no
answered commit. A crash of the host can lose the last ones. A read sees a commit before its log
write. A commit that a standby must confirm (`--sync-slot`) still waits for its sync and receipt.
On TechEmpower /updates (20 rows a request) it raised GizDB from 5,485 to 6,442 req/s, p99 from
171 to 65 ms (a build host, 10/17).

### Durability failures stop the engine

A failed write or sync is not tried again. After a failed fsync, the kernel can drop the dirty
pages and report them as clean (the fsyncgate of PostgreSQL). Thus the engine refuses more work
until it is opened again and has recovered from the log.

### Bounded raw replication chunks

- `Log.chunk` and `chunkReaderWithResources` share the lookup of retained files and the
  canonical range Reader.
- A request stays within its byte credit and one physical file, also inside a large frame.
- The scoped sender keeps its Reader until the outgoing bytes are released.
- Ordered rotation cuts use binary search. Invalid signed frame offsets are refused before a read.
- A transport cut alone does not certify a verified, applied or durable commit. The receiver
  must establish those boundaries through the same WAL checksum and Engine publication owner.

### Bounded base backup and PITR copy

Restore:

- `restoreWithResources` parses the bounded UTF-8 completion label while its buffers stay
  admitted. Its replay coordinate must agree with the checksummed MANIFEST. It verifies a
  complete WAL frame boundary before it writes the target. Invalid metadata cannot move or
  bypass the PITR cut.
- It uses the same restore owner as `restore`, the `ReadResources` of the caller, and the WAL
  `FrameScanner` for checksum validation.
- The exact-range copier of `fs/disk.giz` copies file bodies. No backup segment or
  complete WAL body is collected in memory.
- A short read or write continues. An I/O failure stays an error. Every owned handle and window
  is released before the return.
- A restore still requires a destination that is empty at the start and exclusive. Partial
  files of a failed restore are never reported as recovered.
- Replay coordinates in the label that are missing, malformed, too large, negative or duplicated
  refuse the restore before any destination file is copied. They never become position zero.

Backup:

- `Engine.backup` uses the same copier with a 64 KiB window. `backupWithResources` uses the Store
  admission of the caller.
- `backupWithContext` checkpoints through the actor's storage first, then copies the segment files
  and the logs; evicted segments are copied from the same immutable files.
- Segment and log bodies are complete before the manifest is copied. `BACKUP_LABEL` certifies completion.
- The base-backup fixture refuses whole-file reads, and verifies independent totals of rows,
  counts and lengths after reopen. It repeats the copy with a 1 KiB window under an admission
  limit of 16 KiB.
- The fixtures for admission denial, partial writes and fsync errors require no completion
  label, returned credits, and unchanged query results on the source. A failed destination does
  not damage the source database.
- A fully published persisted checkpoint uses the same backup owner to copy its primary and end
  roots and its segment data and index files. A stored overlay that is not published refuses
  backup until its canonical checkpoint and replay with context finish.

Restore with context:

- `restoreWithContext` copies through the same restore owner and opens through
  `openWithContext`. It returns the canonical Engine under the actor context of the caller.
- The cold-root fixture checks the visibility of updates and deletes, and historical point
  lookup, without a rebuild of resident rows or primary maps.
- `restoreRecoveryWithContext` gives an explicit canonical resource policy and calls
  `openRecoveryWithContext` through the same format dispatcher. A cold archived tail then uses
  the writable WAL replay and the MVCC publication owner. A replay that is not supported is
  refused. It does not change to resident hydration.

WAL archive:

- Closed WAL archival uses the atomic range publisher of the Store. It does not read a full file into memory.
- `archiveLogsWithResources` admits its copy window on the same ledger of the caller. Retention
  moves only after a successful publication and cleanup of handles and windows.
- `archiveNowWithContext` first flushes the pending WAL of the actor through `syncWithStorage`. Then it uses the same
  publisher and ledger. The path without context cannot flush pending pages that an actor owns.
- The archive fixtures check bounded byte-for-byte copy, no work for a cut that is already
  archived, and that the previous complete file and retention cut stay after an admission
  refusal or a failure before publication.
- A failed directory sync after a rename is uncertain durability. The new complete file can
  exist, but the retention cut does not move.

The point-in-time SQL fixture adds a source that refuses whole-file reads, and a payload of
256 KiB under a copy admission limit of 16 KiB. This tests the geometry of the copy, not the
capacity of the full database.

> **Note:** Directory and log metadata still increase with the number of files. The final legacy
> `openOn` recovery keeps resident tables and indexes. These APIs do not certify bounded total
> RSS, native tables larger than RAM, or a completed implementation of mutable online backup.

### Group commit and checkpoint

- **Group commit.** Commits that arrive while a sync runs share the next sync. A transaction is
  acknowledged only after the sync that covers it.
- **Checkpoint.**
  1. Start a new log file at the current LSN.
  2. Write the delta as segment files.
  3. Replace the manifest (staged file, sync, rename, directory sync). The manifest names the
     segments, their deletion state and that LSN.
  4. Only then, remove the older log files.

  A crash at any point replays from the LSN in the manifest that survived. Replay is idempotent by LSN.
- **Why not a B-tree with a WAL (SQLite, PostgreSQL).** Pages that are updated in place need a
  WAL also, and write each change two times. A log plus immutable segments writes one time to each.
- Replication sends these same records. A change feed is a reader of them.

### Standby

**Built**: `db/pg/replica.giz`. Start it with `gizdb --primary host:port` and the password of the primary.

Copy and follow:

1. An empty directory is copied from a checkpoint of the primary: its manifest and the segment
   files that it names. The primary keeps them until the copy is done. The manifest is written
   last, so an interrupted copy starts again.
2. Then the log of the primary streams from that LSN. It is applied as recovery applies it
   (`Engine.replay`, which both use) and appended. Thus the log of the standby is the log of the
   primary, byte for byte, and a restart continues where its log ends.

Slots:

- The standby is an ordinary client of the primary (REPLICATION statements, superusers only).
- A request with nothing to send waits for the next commit, as `changes()` does.
- A request carries the durable end of the standby to its **slot**. The primary keeps its log
  from the slot across checkpoints and restarts (the file SLOTS, as the replication slots of
  PostgreSQL). Thus a standby that restarts or was away continues.
- `REPLICATION SLOTS` lists the slots. `REPLICATION DROP n` frees the log that a slot held for a
  standby that will not return.

Reads, writes and promotion:

- Reads on the standby see each commit shortly after the primary acknowledges it (asynchronous, RPO > 0).
- Writes fail with 25006 until a superuser runs `SELECT pg_promote()`. This must be the only,
  unconditional autocommit command.
- Ordinary users receive 42501. Transaction blocks receive 25001. Conditional, subquery and
  routine calls are not supported, so they cause no side effect at bind time.
- The authoritative engine forks its branch, and atomically replaces and syncs
  `REPLICATION_HISTORY`, before it acknowledges promotion or permits writes.
- A restart with the original `--primary` command keeps the promoted role.
- An uncertain sync of the role file stops follow and writes, and requires a restart of the
  process. The restart reads the durable role again. Ordinary automatic storage retries cannot
  continue the old role against a new branch file.
- `pg_is_in_recovery()` tells which role the authoritative server has.

Branch identity:

- Each live `REPLICATION LOG lsn durable slot 'history'` request checks its expected random
  branch identity before it moves a slot or returns bytes.
- Promotion assigns a new identity. Two conflicting histories at an equal LSN are refused, also
  on a client that is already open.
- `REPLICATION STATUS` reports the branch, the durable LSN (NULL when storage is uncertain), the
  committed timestamp, the recovery role and the latest replication or storage error.
- Lag is measured by a comparison of the durable positions of a primary and its standby. It is
  not inferred from row counts.

### Designated synchronous receipt

This is opt-in.

1. Provision a standby base asynchronously.
2. Restart its primary with `--sync-slot ID --sync-timeout 5s`.

Rules:

- Actual commit replies wait for the local WAL sync, and for a new LOG acknowledgment with a
  matching history of the fsynced receive of the designated standby.
- BASE and the persisted SLOTS retention positions never certify receipt. Reopen and promotion
  clear live confirmations.
- STATUS appends `sync_slot`, `confirmed_lsn` and `sync_waiters`.
- Missing slots, invalid options, and a drop of the selected slot are refused.
- On timeout, FATAL08007 states that the transaction is committed and durable locally, but that
  remote receipt is not confirmed. Then the server disconnects. Examine the state before you try
  again. Do not assume a rollback.
- Writable recovery syncs the recovered tail before it publishes a durable position. Failed sync
  or replay reopens close their log handles. Inspection stays read-only.

Tests: local process tests with four loops cover absence, stale and reopened confirmations,
Flush and Sync, the exact acknowledged rows after a SIGKILL of the primary, a standby restart
with the upstream offline, and durable promotion.

> **Warning:** This trusts the superuser receipt of one designated standby, without consensus or
> automatic failover. The new writes of a promoted node are asynchronous by default. Provision
> and explicitly select its new downstream before you claim synchronous durability for them.

### Divergence and rejoin

- A standby history that is missing or malformed fails closed. Standbys from before the history
  file require a new base.
- A divergent old primary, and a downstream standby whose upstream was promoted, rejoin through
  an **empty directory**. The old directory stays for reconciliation.
- There is no rewind and no automatic adoption of a branch. Fence the old primary before
  promotion. After a permanent promotion, explicitly drop its obsolete upstream slot when that
  primary is reachable.
- Segment copies keep a maximum of one decoded chunk of 4 MiB. They do not buffer a complete segment.
- An interrupted base publishes no MANIFEST. A retry replaces partial files, and releases its
  recorded abandoned slot only on the same branch. A crash before the slot is recorded can leave
  an orphan. `REPLICATION SLOTS` shows it for explicit cleanup.

### Replication tests and endurance

Local process regressions (`tests/db/replication`) exercise actual SIGKILL, live catch-up,
primary and replica restart, divergence at an equal LSN, promotion authority and persistence,
restart of a partial copy, rejoin with a new base, and exact ordered rows with concurrent ingest
and analytical readers.

`bench/db/reliability/replica.giz [milliseconds]` runs one continuous disposable cluster, for fifteen
minutes by default. It has a fixed rolling dataset of 16,384 rows, repeated crashes,
generation-aware row checks, bounded latency histograms, durable lag and resource samples.

| Run | Result |
| --- | --- |
| Preflight, 20,840 ms | Passed: 982 acknowledged transactions, exact final rows, fenced promotion and restart. |
| First fifteen-minute attempt | Failed after a progress sample at 437,044 ms, when a replica restart took longer than the 20 s readiness limit. Its final oracle did not run. Its binary and data were not kept. |
| Later preserved diagnostic run, with the pre-Flush server `937b0b4` | Passed: 900,104 ms, 118,021 acknowledged transactions, 13 scheduled replica crashes, two primary crashes and 77 exact barriers, with final fenced promotion and durable restart. |

The measurements are in `bench/db/README.md`. The original timeout is not explained.

Startup log replay builds the optimizer statistics one time, from the final recovered state.
Queryable commits and live replication keep ordinary resampling. A bounded probe of a retained
copy verified exact cells through replay, checkpoint and reopen. It counted 458 startup scans
that became one, and confirmed checkpoint reclamation (details in `bench/db/README.md`).

**Not supported:** automatic failover based on consensus, acceptance across hosts and after power
loss, and sustained synchronous ingest performance. The local tests of designated receipt do not prove these.

## 3. Version and transactions

**Built** (`db/core/engine.giz`, `db/core/table.giz`), except hybrid logical clocks, which wait for the cluster.

- A row version is visible to a snapshot `s` when `begin <= s < end`. `end` is infinity while
  the version is current.
- An update or delete sets the `end` of the current version to the commit timestamp. An update
  also appends a new version.
- Deletion state is a column of `end` timestamps for each segment row. It is persisted at
  checkpoint and built again from the log after the checkpoint. Thus segment files never change.

### Writers and readers

- **One writer applies commits in series.** Readers read snapshots and never wait.
- **Transactions are optimistic** (the design of FoundationDB).
  - A transaction reads at a snapshot and buffers its writes. Its own reads see them. The buffer
    is indexed by key and unique value, so a read back costs one probe at any size.
  - It records what it read: keys, and the tables that it scanned.
  - Commit is one serial step of the writer. If a commit after the snapshot wrote something that
    the transaction read, the commit fails with a conflict, and the caller tries again. If not,
    its writes are applied with the next timestamp and logged.
  - This is serializable. It holds no locks while the client thinks. It agrees with how the Giz
    app server works: one transaction for each HTTP request, with awaits between its statements,
    committed at the end (`platform/server/src/common/api.interceptor.ts`).
  - A transaction given as a function runs inside the writer and cannot conflict.

### Isolation levels

`Tx.isolation` (`tests/db/isolation.giz`) gives the isolation levels that a client of each engine expects.

| Session | Default |
| --- | --- |
| PostgreSQL, Oracle, SQL Server | READ COMMITTED |
| SQLite | Serializable |

- BEGIN ISOLATION LEVEL, SET TRANSACTION and SET SESSION CHARACTERISTICS select the level.
- READ COMMITTED reads a new snapshot at each statement. REPEATABLE READ (and SNAPSHOT of SQL
  Server) keeps one snapshot.
- Both validate at commit only the rows that they write, the unique entries that they use, and
  the foreign-key checks. Thus readers never cause conflicts. A commit with no commit after its snapshot looks
  none of them up: none can have changed (`Engine.validateReadsIn`).
- The changed rows of a statement, and SELECT ... FOR UPDATE, take row locks (`Engine.lockRows`,
  NOWAIT, SKIP LOCKED).
  - A second writer of the row waits. Then it runs its statement again on the new snapshot (no lost update).
  - A READ COMMITTED write, and SELECT ... FOR UPDATE, read the newest commit, durable or not: a commit frees its
    row locks before its sync (early lock release). The answer that ends the work waits for the sync of each commit
    it read: the statement's own when it runs alone, else its transaction's COMMIT or ROLLBACK (`Client.seen`).
  - REPEATABLE READ fails with 40001. A cycle of waits fails with 40P01.
- A statement alone checks locks only while a transaction holds one.

### Foreign keys

Foreign keys are true at each statement (`Fk` in `db/sql/catalog/keys.giz`).

- A written child row must name a parent row that exists (23503).
- A delete or change of a referenced parent row restricts, cascades, or sets null or the default,
  inside the same transaction.
- The checks are reads. Thus a child committed against a parent that a different transaction
  deletes conflicts at the later commit. It does not become an orphan.
- **Not supported:** DEFERRABLE keys. They are checked at each statement, not at COMMIT.

### Named locks

Named locks wait (as `pg_advisory_xact_lock` of PostgreSQL).

- The second taker waits until the holder ends. A wait that closes a cycle fails with 40P01.
- When a transaction gets a lock, it validates what it read until then, and moves its snapshot
  to the latest commit. Thus its next work sees the writes of the previous holder, and does not
  conflict with them.
- Fifty concurrent transactions each lock their own chat, read it through an index, and append
  to one feed under a shared lock. All commit without a retry. `tests/db/pg.giz` holds the case
  with two transactions.

### Timestamps and old versions

- Timestamps are hybrid logical clocks (physical milliseconds and a counter). Thus the logs of
  one node need no format change for the cluster.
- Old versions are kept until no open snapshot needs them. Then compaction drops them.

## 4. Segment: one storage format

**Built** (`db/core/segment.giz`, `db/core/table.giz`), except the quantized vector copy. A table is a list of
segments and one delta.

### Delta and segments

- The **delta** holds recent versions as rows. Appends and point reads are cheap.
- A **segment** holds a maximum of a fixed number of rows, as columns, in the order in which the
  delta flushed them.
- Each column is divided into blocks of 4096 values, with a zone map (min, max, nulls) for each block.
- A scan reads only the blocks that a `column op constant` condition can match:
  - integers exactly;
  - floats, without trust in a bound that NaN or rounding could cross;
  - text by its position in the sorted dictionary;
  - decimals of one scale by their coefficients.
  - `<>` skips a block that holds only its constant (integers and text). `pg_stat_gizdb.zone_skipped_rows` counts
    the rows that the zone maps skip.
  - A cold text column against `''` reads only the lengths of its dictionary, not its strings.
  - When one condition rules out every block of a part, the others are not checked.
- `min` and `max` of an integer or date column over a whole table, with no condition, come from the zone maps alone
  when every part is a whole segment whose rows are all visible.

A segment file (version 3, `GIZSEG03`) is a sealed head and one sealed block for each column. The head holds
the row count, the `begin` stamps, and for each column the offset and length of its block and its zone maps. Thus an
open reads only the heads, and a query reads only the blocks of the columns that it names (`ColdHome.column`).

Encodings in a block:

| Values | Encoding |
| --- | --- |
| Integers, decimal coefficients, text codes, dictionary lengths | Frame of reference: frames of 2048 values, each its least value and the offsets bit packed (DuckDB's bitpacking FOR mode). A second column of high words when a decimal needs 128 bits. |
| Integers of few runs | Runs: each value and its length, bit packed, when there are at most a sixteenth as many runs as values (DuckDB's RLE). |
| Floats | ALP (DuckDB's): each double as an integer over a power of ten, bit packed, with the values that do not come back exactly kept whole, when that is smaller. Else f64. |
| Strings | A sorted dictionary of codes. A dictionary of 32 KB or more has its bytes FSST compressed (a table of up to 255 symbols of 1 to 8 bytes) when that halves them. |
| Vectors | f32 rows |
| NULLs | One bit for each row, or one byte when no value is NULL |

A version 1 file (`GIZSEG01`, one sealed blob of every column) still opens, whole and resident. It is not in the
files that a checkpoint names, so the next checkpoint (or eviction) writes it again as version 3.


A primary-key lookup goes to the delta, then to the key index of the segments (§5), newest first.

### Compaction

**Compaction** merges small segments into large segments, and drops versions that no snapshot
needs. It builds the derived structures of the result again. It runs inside a checkpoint, in
three steps (`db/core/engine.giz`):

1. The writer plans it: a new log file, the deltas flushed, and the segments and ends of the
   tables taken as values.
2. A thread of its own computes it: merges, segment encodings and the manifest. `runCheckpoint`
   reads only the plan. Commits continue meanwhile.
3. The writer finishes it: files, manifest, log trim.

- The merged segment then replaces its inputs in memory, with the ends written meanwhile and the
  index entries moved (`Table.install`). It does not do this when a DDL came between.
- A plan older than the manifest on disk is dropped (`tests/db/checkpoint.giz`).
- The flush of a delta into a segment still runs on the writer.

This gives point reads at the cost of an index probe, and scans at column speed, from one copy.

**Why not two copies, rows for OLTP and columns for OLAP.** Two copies double the writes and need
a replication path between them. The delta is the row copy. It is bounded and lives for a short time.

## 5. Derived structures and watermarks

A derived structure is a pure function of a segment (and of the delta, incrementally). It is
stored next to the segment and built again by compaction. The key index, the zone maps and the
inverted index (`db/core/inverted.giz`) and the vector index (`db/core/vector.giz`) are **Built**.

| Structure | For | Mechanism |
| --- | --- | --- |
| Key index | Point reads, upserts | A hash of the encoded key to (segment, row), for current versions. |
| Btree index | Equality on a prefix, ranges, ORDER BY ... LIMIT | A B+tree (`fs/ordered.giz`) of order-preserving keys for each table. One entry for each version. Every write keeps it current. |
| Zone maps | Scans | Min and max for each block of each column. |
| Inverted index | Full text | For each text column: a term dictionary, and postings of row numbers and term frequencies in blocks of 128, with the maximum BM25 contribution of the block. |
| Vector index | Nearest neighbors | For each vector column: an HNSW graph over the rows of the segment, walked on RaBitQ (or SQ8) codes. An exact scan below the size where the scan is faster. A cold segment keeps the codes (and by `graph`, the links) in memory and reads vectors from its file. |

Each derived structure records its **watermark**: the commit that it shows.

- A plan that uses a structure reads the commits after its watermark from the delta, or waits.
  For the structures above, the read from the delta is always possible, because the delta is
  searched exactly.
- Results never claim a freshness that they do not have.
- Asynchronous structures (embeddings that a model computes, materialized views over sources)
  report their watermark with the result.

### Live materialized views

`CREATE LIVE MATERIALIZED VIEW v AS SELECT keys, aggregates FROM t [WHERE ...] GROUP BY keys [HAVING ...]` makes a
table of the query's rows. Each write to `t` changes `v` in the same transaction (`Conn.maintainLive` in
`db/sql/liveview.giz`). Its watermark is always the reader's snapshot: a commit shows `t` and `v` together, and a
rollback undoes both.

- The query reads one table. Each GROUP BY expression is also a select item. Without GROUP BY, the view is one row.
- A statement that only inserts adds the partial aggregates of its new rows to their groups, when every other item is
  `count`, `sum`, `min` or `max` and there is no HAVING.
- Any other write computes again the groups that its old and new rows touch. Past 1000 groups, or without GROUP BY,
  it computes the whole view again.
- The write runs on the row path: a table with a live view takes no bulk load or bulk rewrite.
- TRUNCATE of `t` computes the view again. `REFRESH MATERIALIZED VIEW v` does too.
- Only maintenance writes `v` (42809). DROP TABLE `t` needs CASCADE, which drops `v`. `DROP MATERIALIZED VIEW v`
  drops it.
- Two transactions that write rows of the same group wait for each other on the view's row, as under READ COMMITTED.
- A plain `CREATE MATERIALIZED VIEW` is a view: it is always fresh, and REFRESH does nothing.
## 6. Plan: one algebra

**Built** (`db/sql/`): parse (the spellings of SQLite, PostgreSQL, MySQL, SQL Server and
Oracle), bind, optimize, execute, and the session that runs statements in transactions.

### Bind

`bind.giz` resolves names, folds aggregates and windows into their operators, and changes
subqueries into plans.

- An uncorrelated subquery runs one time.
- A correlated subquery becomes APPLY, with a cache by outer values. It goes under the
  aggregate when the select groups, and under the sort when ORDER BY reads it.
- A subquery written two times is one plan. Thus an item that repeats a GROUP BY key reads the key.
- Views and CTEs expand in place.
- WITH RECURSIVE binds its first SELECT as the anchor, and the remainder as a step that reads the
  rows of the last turn (RECURSE over a WORK leaf). It runs until a turn adds no rows. UNION drops
  rows that it already produced. Each turn checks statement_timeout.

### Optimize

`optimize.giz` does these steps:

- It pushes predicates through projections, joins and aggregate keys.
- It extracts equi-join keys. This includes a conjunct that every branch of an OR holds.
- It unnests correlated scalar and EXISTS subqueries into joins.
- It orders inner-join chains by the lowest estimated cost: dynamic programming over subsets,
  and greedy after DP_INPUTS. Each step hashes the smaller side.
- It changes equality on a primary key or a full index into a lookup. The value may be a parameter or a library function
  of constants and parameters that is not volatile (`id = MOD($1, 1000) + 1`).

Estimates:

- A join on the unique key of a side is estimated as a foreign key: the rows of the other side
  times the fraction of that table that is left.
- A key of several columns has a maximum of as many values as its side has rows.
- A join of few rows with a grouped aggregate on its group keys first semi-joins the input of
  the aggregate with their keys. Thus the average for each part in TPC-H Q17 reads only the
  joined parts.

Index ranges:

- A selective comparison on a btree index column (after equality on its leading columns), or
  ORDER BY its columns with LIMIT, becomes a walk of the key range of the index (RANGE, forward
  or descending).
- Every condition is still checked on the rows that the walk returns.
- The walk is recorded as a read of its key range.

`EXPLAIN` prints the plan with row estimates.

### Execution receipts and budgets

Finalized invocation profile slots accept measured execution receipts through `Exec.recordOperator`:

- emitted rows and the invocation count;
- elapsed milliseconds (optional);
- actual work rows, kernel calls and polls;
- successful spill read and write bytes (optional);
- cancellation events.

Rules:

- Receipts are deltas that the execution owner publishes one time. A record never executes a plan.
- Optional counters keep their sample coverage. Thus missing or partial instrumentation is not a
  measured zero or a complete operator total.
- Spill byte receipts give successful physical transfer lengths, with framing. They do not give
  temporary capacity, payload estimates or process I/O.
- The invocation owner serializes trace mutation.
- Stored-graph reports and their physical I/O instrumentation need their receiver integration.
  This API alone does not prove them.

`Exec.partitionExecutionBudget` divides one state allowance among the declared graph owners and
their exact live phases.

- Connected prepared SORT sources and their receiver receive the same budget. They keep the
  exact admission checks of the source.
- Simultaneous minimum grants must fit before spare capacity is shared.
- Disjoint phases can use an allowance again only after their resource owners close.
- Row, output and work limits for each poll stay the configured limits.
- The fixed planning credit and the declared overlap that is not state are included in the computed peak.
- This peak is a conservative grant limit. It is not observed RSS or elapsed performance.
- Runtime enforcement requires the execution fence of the same issuer in `ResourceContext`. A
  phase schedule of a caller that is not complete is not proof of bounded execution.

### Execute

`exec.giz` processes **morsels of 16384 rows as columns**.

Operators:

- Pipelines (scan, filter, project, probe) run in parallel by morsel (`Task.parallel`).
- **Hash join** builds a table divided into 64 partitions in parallel: inner, left, full, semi,
  and anti with NOT IN semantics.
  - It uses only the columns read above it, its keys and its condition.
  - It filters candidate pairs 65,536 at a time. Several integer keys are packed into one exact number.
  - A semi or anti join without a residual condition builds only key membership. It permits
    duplicate integer keys, and needs no concatenated payloads and candidate pairs.
  - A selective semi join moves below plain column projections and repeatable inner joins when
    all its keys read one input. It keeps the multiplicity of that input. Outer joins and
    computed projections stop it.
- **Hash aggregate** pre-aggregates in each task and merges by partition. Counts and sums of
  integers, floats and decimals of one scale use arrays for each group.
  - **Typed groups** (`typedAggregate`) take a GROUP BY whose keys are integers, booleans, dates,
    times or text, and whose aggregates count, sum, average, take an extreme, or take a DISTINCT of
    one integer column. A key is a 64-bit word: a text key is its string's hash, which a batch computes once for each
    dictionary entry its rows use; a group keeps its strings, which decide when the words agree, and the merge moves
    them with the group (no map of strings). The groups are one open-addressing table of words, whose slots
    keep 23 bits of the group's hash, and each aggregate keeps one array of state for each group (DuckDB's
    `aggregate_hashtable.cpp`). A sum that passes an i64 keeps that group in the row loop's exact state. A key that is
    another key plus, minus or times constants (`ClientIP - 1`), or equal to an earlier key, is not hashed: the finish
    computes it from that key's column for every group (`typedKeys`).
  - A group costs its key words, a hash, two slots and 9 to 41 bytes for each aggregate: an aggregate makes only the
    arrays of the values it meets (`TypedAgg.made`), so a NUMERIC sum takes 17. The tasks hold at most one entry for
    each input row, and the merge holds each group once more.
  - A task's table that reaches 16,384 groups (`TYPED_LOCAL_GROUPS`) doubles its limit when its groups will reduce the
    task's rows (`typedGrows`): the rows it took to fill give the keys' distinct values as if drawn at random
    (`typedDistinct`), and from them the groups of the task's rows. A table past three quarters of a core's share of the
    last-level cache (`gizlang:machine` `cacheShare`: 4 MB on a Ryzen 5900X, 32 MB for each 6 cores; 4 MB when
    unknown) grows only for eight rows a group or more. Otherwise the table goes to its partitions (a piece). When its groups
    held fewer than two rows each, the task's later rows go to the partitions as they are, their keys and arguments
    only (`TypedGroups.spread`), and each partition's merge folds them (`spreadRel`, `foldSpread`).
    A task joins the rows of its morsels into batches of 65,536 rows before it splits them (`SPREAD_ROWS`): split per morsel,
    TPC-H Q15's 3,000 morsels made half a million columns of a few dozen values, and freeing them was half the query.
    `pg_stat_gizdb.spread_rows` counts these rows. The probe hashes 256 rows at a time and reads their slots ahead.
  - A GROUP BY on one integer key meets rows stored in key order (TPC-H Q18's `l_orderkey`): a task's groups are then
    made in ascending key order by appending, with no hash and no slot (`TypedGroups.ordered`; a batch out of order
    puts the groups in the table and goes on as above). The tasks read key ranges of their own, so only a key that
    crosses two tasks can be in two of them: `orderedJoin` merges those groups and the groups need no partition, no
    merge table and no copy. A task's ordered groups may hold the state less the merge's half. When the tasks' keys
    do not ascend from one to the next, the groups are partitioned as above (more passes when a copy would not
    fit). `pg_stat_gizdb.ordered_groups` counts the groups finished this way.
  - The typed groups check their bytes per morsel against the statement's state (`Ctx.stateBytes`). A task that spreads
    foresees its bytes at its end, and may hold the state less the groups its merge will make.
    Past it they first take, once, what is left of the memory no credit counts (`Engine.grant`: a quarter of the process's,
    shared by every loop) and give it back at their end. Past that too they read their input again in passes: each pass folds only the rows whose keys hash to it (a text key by
    its string), so it holds a share of the groups, and the passes are enough for each to hold half the state, as
    DuckDB's radix partitions do without the spill (the input is columns, read again with no write). A top-k keeps its
    first groups of each pass (`pg_stat_gizdb.group_passes` counts the passes past the first). Past 64 passes they fail
    with `HASH_OVERFLOW`, and the statement runs again on the
    estimates, which sort when the groups do not fit.
    Beside the state, the run holds the columns the cells keep (the tables budget), one window read past them (at most half
    that budget, `resolveParts`) and about 1.3 MB a task (its morsel's rows, their hashes and its table). None of these
    grows with the input or the groups: groupspill at a 4 MiB state needs a 56 MB process limit on one core for 1M rows
    and 72 MB for 4M rows.
- **Stored statements** run a plan of scans, filters, projections, hash joins, aggregates, sorts and LIMIT
  whole on this executor (a join only below an aggregate or a top-k sort, whose rows stream into it; a build past half
  the statement's state plans again with `HASH_OVERFLOW`) (a `STORED_RESIDENT` node), when its estimated state fits half of the
  statement's. In the first (optimistic) build, the state of typed groups counts as zero, because
  they check it as they grow. EXPLAIN ANALYZE shows the rows of each node of that plan.
- **Sort** orders typed keys (raw integers, decimal coefficients, text by dictionary rank) in
  parallel chunks merged in pairs. Under LIMIT, it keeps a top-k cut. A top-k whose first key is a count of the typed
  groups below it (`GROUP BY ... ORDER BY count(*) DESC LIMIT 10`) has each partition of groups finish only its first
  groups by that count, and every group tied with the last of them when more keys follow (`aggregateTop`): the other
  groups' aggregates (an AVG's NUMERIC each) are never computed. A LIMIT with no ORDER BY over typed groups (through
  projections, no HAVING) finishes any LIMIT groups of each partition, and its passes stop once it has them
  (`aggregateAny`).
- A scan's first filter reads its own columns for every row of a morsel, and the other columns only for the rows it
  keeps.
- A filter drops the columns that only it reads.
- With no GROUP BY, sums of one integer x plus constants (`sum(x + 1)`, `sum(x - 3)`, `sum(x)`) share one sum, count, minimum
  and maximum of x, and each answer is sum(x) + k * count(x), as DuckDB's SumRewriter (`offsetSums`). When x + k can pass
  its width for x's least or greatest value, or a total passes an i64, the sums run as written and fail or widen as they do.
- **Compiled scans** (`compiled.giz`): a scan's filter, and with no GROUP BY its aggregates (count, sum, avg, min and
  max) too, become one Gizlang function over a part's stored columns: one loop that reads each row once (Umbra's
  pipelines). Its text is generated from the plan (integer, date, time, NUMERIC and text-code comparisons, integer,
  NUMERIC and double arithmetic, AND, OR, NOT, IS NULL, IN), with its constant parts evaluated once (vm.giz folded).
  NUMERIC columns of one scale of up to 18 digits are their coefficients; the Oracle dialect, whose NUMBER trims each
  value, keeps NUMERIC arithmetic on the interpreter. `gizlang:functions` compiles the text once per sha256 in the background, at
  -O2, into `GIZLANG_CACHE/functions`. Until the unit is loaded the statement runs on the interpreter, which stays the
  reference: a part whose columns are of other kinds, or a run whose integer arithmetic passes its width, runs there
  too and fails as it does. Constants are the run's params, so a unit serves its shape with any constants. An installed
  toolchain's `bin/gizdb` runs the server with its version's CLI as `GIZLANG_CLI` (launcher.giz databaseEnvironment), so
  it compiles each new shape as a build does. A program with no `GIZLANG_CLI` (a server-only release, which holds no
  compiler) compiles none: it links the units its warm-up met (`build --warmup`, with
  bench/db/engine/warm.giz running the TPC-H shapes), and any other shape runs on the interpreter's bit filters (pred.giz
  compareBits: integer, date, time, NUMERIC, float and text columns against constants).
  A GROUP BY of distinct text columns over a scan compiles too (compiled.giz groupKernel): the filter, the arguments'
  arithmetic and the aggregates in one loop, a group a slot of its keys' dictionary codes (at most 4,096 slots, as
  DuckDB's perfect hash aggregate), whose run merges into the typed groups (TypedGroups.foldSlots).
  `pg_stat_gizdb.group_kernel_runs` counts these runs.
  The filters and the joins that follow a stored scan compile as one loop too (compiled.giz chainKernel, a pipeline's
  `Source.chain`): a row is read once, probes each inner, semi or anti join of one integer key (or a packed key of
  several) through its rank index or its key tables, and keeps the build rows in registers; the columns of a
  build that a later key, condition or residual reads are loaded from the build row. Only the rows that pass every link
  are written, each as its scan row and the build row of each inner join, and the interpreter gathers the columns of
  those rows alone and goes on with the steps after the last link the loop covers. How a build finds its keys (rank index,
  Bloom filter, tables) is a param, so one unit serves a join shape at any scale. TPC-H Q5 probed 60M
  lineitem rows through four joins on the interpreter, each a pass over the rows and a gather of every column.
  `pg_stat_gizdb.chain_kernels` counts the pipelines that ran with a loaded loop.
- A scan is bound by memory, not by the cores (TPC-H Q6 reads four columns of 60M rows: 1.9 GB as 64-bit values, 36 GB/s on
  the bench host). A kernel reads a scan's integer and NUMERIC columns as **lanes** of 8, 16 or 32 bits, the narrowest that
  the zone maps of every part keep their values within (`Part.laneBits`, `Segment.laneBits`; a cold segment only, which a segment a checkpoint wrote becomes at once, its columns kept in
  its cells, as on reopen: `Engine.evictCold`): a
  16-bit lane reads a quarter of the 64-bit bytes, an 8-bit one an eighth (a flag, a small code). A lane is a copy of the column built once on every core before the
  morsels run (`ColdHome.keepLane`, `ColdHome.keepShort`, `ColdHome.keepTiny`), kept in the same cells as the columns,
  counted in the same budget and released with them. The kernel's shape names which columns it reads as lanes, so one
  unit serves a scan at any scale; a part whose lane is missing runs on the interpreter. A core running a morsel only reads
  lanes, which `Compiled.lanes` carries. `pg_stat_gizdb.lane_kernels` counts the pipelines that ran with lanes.
- `LIKE` of a text column against a constant pattern without `_`, an escape or case folding is a kernel test too: before the
  morsels run, each part's dictionary is matched once in dictionary order, on every core, in tasks of 65,536 entries
  (`pipeline.giz dictsOf`), and a row then reads one bit by its code (`Compiled.dicts`). A row-by-row match read a string
  object at a random place for each row: 200 ns a row on TPC-H's 2M part names (Q9, Q20) and on its 15M order comments (Q13).
  A part whose morsels read under a quarter of its rows (a top-N's zone cut) matches only the entries those rows hold.
- An integer `IN` list and `IN` a subquery's integers run over the batch as a binary search (`vm.giz memberKernel`);
  `COUNT(DISTINCT integer)` with keys is one operator (typed groups keep (group, value) pairs: a task in a set of 65,536
  slots that it empties into a list when full, a partition in one set sized for all the pairs it takes); `COUNT(DISTINCT integer)`
  without keys sends each key, but a repeat of the row before, to a hash partition, which counts its keys in a set
  (`exec.giz distinctKeys`; a text key sends the strings of each part's codes; `pg_stat_gizdb.distinct_keys`); a top of count(*) by one integer or text key (ORDER BY count(*) LIMIT k, no other
  aggregate) sums each key's runs in such a partition's table and keeps each partition's first k (`exec.giz topCounts`;
  a text key's tasks count each part's dictionary codes and send each code met to its string's partition;
  `pg_stat_gizdb.top_counts`); the COUNT of the other
  side of a LEFT join grouped by the preserved side is the sum of counts a join key (`optimize.giz eagerCounts`).
  A text GROUP BY key that its table's primary key fixes (TPC-H Q10's `c_name`, `c_address`, `c_phone`, `c_comment`) is not grouped: under
  ORDER BY ... LIMIT the groups by the other keys give their top rows, and a join on the primary key reads the text
  for those rows only (`optimize.giz lateKeys`). A GROUP BY key that the other keys fix, a constant or an expression of
  integer key columns (ClickBench's `GROUP BY 1, URL` and `ClientIP - 1` beside `ClientIP`), is not grouped either: a
  projection above the aggregate computes it for each group (`optimize.giz impliedKeys`).
  A WITH query of aggregates or joins that the statement reads twice or more runs once, before its readers, as a subquery of kind
  MATERIAL; each reader scans its batches (`bind.giz Binder.table`, a WORK leaf of kind WORK_SUB): TPC-H Q15 computed its revenue twice.
- Window functions, set operations, LIMIT/OFFSET and generate_series complete the set.
- A transaction reads its own writes.

### Window frames

Window aggregates keep one mergeable state for each row.

- A frame from the start of the partition (a running total) folds each row one time.
- A sliding frame merges O(log n) states of a segment tree in row order.

ROWS frames:

- They use signed preceding and following offsets, also an empty frame.
- SUM frames whose integer sign totals can overflow fold their rows in order. This keeps the
  intermediate-overflow errors of the aggregate, and does not refuse a safe selected frame.
- ROWS offsets are clamped before the addition. Thus the largest i64 bound cannot overflow.

RANGE frames:

- They support unbounded and current-row peer frames, and numeric distances over one ORDER BY
  expression. This includes ascending and descending order, NULL peers, and fractional decimal
  and float offsets.
- Each peer group shares a frame. Binary searches over its representatives find the boundaries by value.
- PostgreSQL integer and decimal thresholds stay exact. This includes values above 2^53, and
  sums larger than the stored range of 40 digits.
- SQLite uses its scalar arithmetic owner for REAL rounding and integer overflow at distance comparisons.
- Offsets are expressions without rows, evaluated one time for each window. NULL, negative, NaN
  and incompatible offset types are errors.
- Timestamp and interval distances, and scalar-subquery offsets, are not supported.

Named WINDOW definitions resolve within their own SELECT, with inherited partition and order
clauses and explicit ROWS frames. Invalid references and inheritance are errors.

VALUES and numeric UNION arms use the numeric common-type conversion of CASE and COALESCE. Thus
mixed integer and float, or integer and decimal, data keeps numeric order.

### Runtime filters and ASOF joins

- An inner or semi hash join on one integer key gives its probe scan the range of the build keys
  as a condition (a runtime filter). Thus zone maps and Parquet statistics skip the data that no
  build key reaches.
- A hash join on integer keys over a range of at most 2^28 values, not sparser than a key in a thousand values (and a
  megavalue), is a rank index (`Built.ranked`): a bit for each value of the range, the keys before each 64-bit word,
  and the build rows in key order (`order`, with `firsts` when a key has several rows). A key's number is its rank
  (the bits before it, by popcount), so a test is a bit and a lookup is two reads, with no hash and no table; keys
  probed in order read all of them in order. TPC-H SF10 Q5 measured 10.8 ns a row on a hash table and 2.3 ns on the
  bits, over 30M rows. The index is built in parallel, one part for each range of words. A build whose keys ascend in
  its row order (a scan of a table in primary-key order) needs no parts: each chunk of its batches sets the bits of its own
  key range, the build rows are the ranks, and there is no `order` (`sortedRank`); equal keys keep the start of each run
  in `firsts`. Other joins keep a map in 64
  partitions, and above 262,144 keys a blocked Bloom filter (16 bits a key, two bits in one word a key) tested before
  it: in a selective join most probe rows miss (StarRocks' runtime filter).
- The inner and semi joins on one integer column of a scan cut its morsels first (`joinFirst`): each tests that column of
  every visible row against its build (rank bits, else range, Bloom filter and map), the scan's filter then reads its
  columns for the rows left, and only the rows that pass are read whole. A test that keeps most of a morsel's first
  1,024 rows is left to its probe.
- A join that a plan computes in several places is computed once before the plan runs, with the columns its places
  read, and each place reads its rows (`sharedJoins`, a WORK leaf). The membership tests of TPC-H Q21 are reduced by
  the rows of the join they test, which was computed three times. `pg_stat_gizdb.shared_joins` counts these joins.
- `t ASOF [LEFT] JOIN q ON t.k = q.k AND t.ts >= q.ts` matches each row with the nearest row of
  its key at or before its time (or after, by the comparison). It uses binary search over the
  right side, grouped by key and sorted by time.
  - Integer clocks that agree and temporal units that agree use exact i64 buffers to order and
    search. Other types keep the SQL comparison.
  - Stable ties select the last right input row for backward searches, and the first one for
    forward searches.
  - Serial and parallel policies share the same probe implementation.

### Statistical aggregates

These aggregates can merge in pieces:

- The covar_pop, covar_samp and corr family of PostgreSQL, and the regr_* family, over one
  co-moment state (Welford updates, Chan merges).
- `ols(y, x1, ...)`: least squares by centered incremental QR. It returns `'{b0,b1,...}'`, or
  NULL when the inputs are collinear.

Rules:

- Correlation and R-squared normalize each centered moment separately. Thus a moment product
  does not overflow for finite large inputs.
- The shared finisher limits correlation to [-1, 1] and R-squared to [0, 1] when centered
  products differ a little through roundoff. This agrees with the mathematical result ranges and
  with the [finisher policy of PostgreSQL](https://github.com/postgres/postgres/blob/master/src/backend/utils/adt/float.c).
- Native covariance-consistency checks still refuse larger violations before the finish.
- The intercept multiplies the finished slope.
- Sample variance and standard deviation use N-1, and give NULL at N < 2. Population forms use N,
  and return zero for one value.

### Native moment state

The native `MomentState` API holds only the count, one or two means and the centered products.
It uses typed nullable f64 rows, signed multiplicities that are not zero, and bounded lossless CRC checkpoints.

- Unit transitions keep their arithmetic order.
- Repeated insertions merge one constant partition. Repeated retractions apply one scaled
  correction of the centered matrix. Matrix work and state size do not increase with the number of copies.
- Population and underflow checks run before signed negation, also for i64 MIN.
- A complete retraction of a repeated row also requires the matching centroid and zero centered
  products. This is not a proof of old-image membership.
- Insertions and partition merges share the co-moment owner. Native partition merges normalize
  population ratios before centered products, to prevent a false intermediate overflow or underflow.
- This explicit native policy has its own history of partition rounding. Ordinary SQL merges
  keep their arithmetic order.
- Result definitions use the variance, stddev, covariance and regression finishers.
- Retractions require old-image attestation from the only membership owner of the caller. Their
  floating rounding history is different from a cold ordered SQL replay. Thus maintained SQL does
  not select this inverse silently.
- Finite-range, covariance-consistency and cancellation checks can refuse removals that are
  mathematically valid, and require an explicit rebuild. They do not guarantee arbitrary accuracy,
  and they do not infer membership from sufficient moments.
- Empty and singleton centered products are zero. NULL rows are skipped. Inputs and results that
  are not finite are explicit native errors.
- The population has a limit of 2^53-1. Byte credit limits the fixed encoded state, not the
  costs of the allocator or RSS.

### Aggregate continuation codecs

Ordinary finite aggregate kernels expose one bounded CRC continuation codec. It is bound to the
trusted aggregate name, the star flag and the arity.

- COUNT, numeric prefix, decimal and Welford, BOOL, BITS, bounded representative Values,
  co-moments, QR and bounded explicit histograms write only their active fields.
- DISTINCT maps and collecting aggregates require external strategies.
- Sequential spill and reload keep the same fold, raw floating bits and integer overflow history.
  They do not make reordered floating partition merges equivalent.
- The caller owns temporary disk ranges and shared resource leases.
- `aggregateKernelSize` is the exact wire credit. `aggregateKernelMatrixBytes` preflights the
  matrix cells before an ordinary step that allocates.
- Neither counts allocator overhead, spare Vec capacity, simultaneous input and output buffers, or process RSS.
- Invalid definitions, shapes, checksums and byte allowances fail explicitly.

The finite grouped cursor stores COUNT, SUM, AVG and TOTAL kernels that the registry admits
separately from generic `Acc` kernels. Small references to plan positions connect them.

- COUNT has no numeric payload. Numeric kernels keep one canonical scalar payload. They do not
  keep unused variance, representative, model or histogram fields.
- The step, finish and continuation owner is used through a transient `Acc` bridge. There is no
  alternative integer-prefix, decimal or floating arithmetic.
- Interleaved generic variance, MIN and BITS, and all output positions, keep their semantics.
- The native running-window `SumCtx` of SQLite stays on its route.
- Shell and overlap credits do not change while that bridge is necessary. No smaller admission
  limits and no gains in physical RSS or latency are claimed.

### Histogram continuation

Histogram continuation stores native boundaries, exact bucket populations, an optional raw
floating sum, the interval and temporality, and the source provenance directly. It has no JSON
shadow and no reconstructed observations.

- Geometry and provenance widths are checked before vector allocation. The native limits stay
  4096 boundaries, 4096 sources and 65536 identity bytes.
- Canonical histogram validation owns the interval, bucket, duplicate-source and sum contracts.
- Shared resource admission must also cover decoded vector capacity, provenance validation maps
  and hash scratch, and simultaneous wire, input and output buffers. The wire allowance alone is
  not that resource limit.
- Reload keeps the sequential histogram merge history. It does not promise rounding of optional
  floating sums that is independent of order, or exact percentiles from buckets.
- Native source fixtures cover reload at each merge, raw sum bits, and malformed widths, UTF-8, checksums and
  definitions.

`aggregateKernelScratchBytes` gives a conservative transient credit for native histograms, from
the wire allowance of the caller. It is capped by the same geometry and provenance limits before
the arithmetic.

- It covers decoded geometry and text slots, growth of the validation map, sorted provenance and
  Value vectors, and the overlap of tuple and hash scratch. Its maximum is 6,561,824 logical bytes.
- It refuses smaller lease allowances of the caller, and kernel strategies that are not supported.
- The executor must get this shared resource lease before it decodes or validates. It must keep
  the lease until the decoded point and the temporary work are released, independently of the
  input and output wire leases.
- A fold that grows must reserve against its configured maximum state-wire allowance, not only
  against the length of the last decoded frame. Incoming histogram parsing and merge operands
  need their own overlapping leases.
- This credit is an explicit conservative admission policy. It is not a measurement of live
  allocation or RSS, and it does not permit the omission of other aggregate, input or output reservations.

### Compact counted keys

Compact counted-key strategies make zero or one key a small payload. They allocate the Ordered
index only for a second distinct key.

- A removal back to one key releases the promoted index in the new version. Immutable earlier
  versions keep their own state.
- COUNT DISTINCT, extrema, exact weighted percentile and MODE use the same canonical key,
  multiplicity, comparison, weighted summaries and codec and cell contracts. There is no second
  tree or checkpoint format.
- `compactKeyReservation` owns the conservative physical credit for one version of this
  representation. The executor adds its membership, receipt, transient and other state credits.
- Logical reservations and local work counters are not measured allocation, RSS or latency.

### OLS

OLS folds centered rows with Givens rotations, and merges triangular factors without a store of observations.

- Its triangular solve equilibrates predictor units by exact powers of two.
- It keeps its successful finite compensated rows.
- Temporary normalized coefficients that are larger than the floating range use mantissa and
  exponent compensation in the same term order. Predictor units are restored before the
  coefficients are materialized.
- Very small independent RHS channels and offdiagonals are not erased by one global scale.
- This is compensation at finite precision. It is not arbitrary precision, and not a guarantee
  for every ill-conditioned design.
- Final coefficients that cannot be represented stay explicit errors.
- Native source fixtures pin wide units, cancellation of large terms and subnormal units.
- The numerical rank test keeps the squared limit of 1e-12.
- Complete input rows must be finite. NULL rows are skipped. A factor or coefficient larger than
  the finite f64 range is an explicit error.

One private compensated linear-combination owner serves triangular back-substitution, intercept
reconstruction, inference loadings and covariance, and point prediction.

- It keeps small terms across cancellation, without a second model or a persistent cache.
- Outside the adaptive triangular solve, terms and partial sums that are not finite stay explicit errors.
- It cannot recover slope information that an ill-conditioned design already lost. It does not
  promise exact arithmetic past the precision of finite f64.
- The private triangular solver returns explicit range errors to fit and inference. Coefficient
  scaling does not change, and gives no finite substitute for an overflow.

Centered QR insertion and merge keep their finite-difference arithmetic. When the subtraction of finite opposite
extremes alone overflows, the owner normalizes mean weights and scales the observations before the subtraction, for the
centered factor. This admits means and factors that can be represented, without raw observations or a second model.
Factors that really overflow still fail.

#### `ols_summary`

`ols_summary(y, x1, ...)` reads the same augmented factor. It returns JSON text with these fields:

| Field | Meaning |
| --- | --- |
| `n` | The number of complete rows. |
| `rank` | The number of coefficients. |
| `df_residual` | `n-rank`. |
| `coefficients` | Intercept first. |
| `rss` | From the residual norm of the factor. Thus it does not subtract two squared totals that are almost equal. |
| `sigma` | `sqrt(RSS/(n-rank))` at positive residual degrees of freedom. |
| `covariance` | The full matrix, intercept first. |
| `std_errors` | Standard errors. |

- Covariance and standard errors use the classical unweighted homoscedastic least-squares estimator.
- Coefficients and standard errors use response units for the intercept, and response/predictor
  units for each slope. Each covariance entry has the product of its two coefficient units.
- At zero residual degrees of freedom, coefficients and RSS stay available. `sigma`,
  `covariance` and `std_errors` are JSON null.
- Both model results return SQL NULL for deficient rank or too few complete rows.
- An RSS or covariance that cannot be represented is an explicit summary error. The result with
  coefficients only does not require a finite RSS.

#### `ols_predict`

`ols_predict(model, x1, ...)` returns a FLOAT point prediction from the intercept-first
coefficients of a summary.

- A NULL model or predictor returns NULL.
- The number of coefficients and the full rank must agree with the given predictors.
  Coefficients and numeric predictors must be finite.
- Compensated summation keeps small terms across cancellation. Products, partial sums or results
  that are not finite are explicit errors.
- Broadcast and dictionary text models are decoded lazily, one time for each distinct entry in
  the batch, after its first complete predictor row. Other model representations use scalar
  decoding for each complete row. No persistent model cache is kept.
- Prediction intervals, formula evaluation, weights, hypothesis tests, and compatibility with
  formulas, packages and the R package ecosystem are outside this contract.

### Percentiles

- Percentiles validate their direct probability, also over an empty or all-NULL input. They
  return NULL for a NULL probability. They interpolate opposite finite extremes without an
  overflow of their difference.
- Literal, parameter, cast, scalar-expression and uncorrelated scalar-subquery probabilities are
  evaluated one time for each group.
- Direct expressions can read grouping keys. Other input columns and nested aggregates are
  refused, also in dialects that permit ordinary bare columns.
- Ordered percentiles and mode keep ASC and DESC, also for discrete ranks and mode ties.

### Bounded external aggregate finalization


#### Ranks

`finishAggregateRanks` uses the percentile probability and interpolation owner. It seeks a
maximum of two non-NULL ranks that the external sorted-input operator gives.

- That operator certifies, before it seeks, the permitted argument families of the full input,
  and the identical comparison, direction and representative order. Reads of selected ranks
  cannot validate input that was not seen.
- The native rank population has a limit of `2^53-1`. It uses inherited floating probability
  arithmetic, not exact rational rank semantics.

Finite cursors:

- A cursor can call `prepareAggregateRanks` one time, and use `aggregateRankRequests` to capture
  zero, one or two selected values across bounded input polls.
- The returned position has the same descending direction and floating interpolation meaning.
  The cursor owns the certification of the complete non-NULL population and the comparison order.
- `finishAggregateRankSelection` requires values in request order, and uses the only percentile
  interpolation owner.
- The callback finisher delegates to this API. It still refuses an invalid first value before it
  reads a second value.
- No sample vector of the population size and no repeated rank scan is necessary.

#### `diskRankGroupedFinite`

The explicit `diskRankGroupedFinite` strategy uses the external SORT and the shared Resources
owner, for one median or percentile aggregate without DISTINCT over a proven stored INT column.

1. It counts and type-certifies each complete group.
2. It visits the sorted run of that group again, to capture the canonical one or two requested ranks.

- Both passes yield after the configured source-row work credit, also for one hot group.
- It keeps only bounded start, end and current reader pages, group keys, and a maximum of two
  selected Values.
- Direct probabilities first admit group columns, constants and parameters. Computed inputs and
  types that are not proved refuse admission.
- NULL, filter and direction rules, and interpolation, stay in their owners.
- The SORT preparation is synchronous. Bounded rank polls do not give bounded constructor latency
  or a scan route for persisted tables.

#### Distinct keys

`finishAggregateDistinctKeys` counts runs of distinct canonical equality keys through a monotone
`AggregateKeyPull` callback (`Result<Option<bytes>, string>`).

- Its only capability owner admits COUNT and count_big, and the exact intrinsic definition of
  approx_count_distinct.
- The external run owner makes Table keys, removes SQL NULL, sorts bytewise, certifies the
  population, and owns IO and input leases and the enforcement of work and deadlines.
- Finite executors can use `beginAggregateDistinctKeys`, `stepAggregateDistinctKey` and
  `finishAggregateDistinctKeyFold` to pause after any input page.
- `KeyFold` keeps only counters, declared limits and the previous opaque key. Refusals occur
  before mutation. Immutable earlier poll versions stay valid.
- The pull wrapper delegates to the same fold. One transient canonical COUNT accumulator is made
  only at finalization.
- No value vector or deduplication Map grows with the population.
- It checks the sort order, the declared key size, the exact end of input, callback errors and the result-wire allowance.
- `aggregateDistinctKeyCredit` reserves a conservative logical overlap of
  `2,048 + 8 * maxKeyBytes` before the pull, which includes both key buffers.
- SUM and AVG DISTINCT do not use this unordered path. They require the restoration of the first
  retained ordinal, to keep the prefix and floating history.

#### Pulled values, MODE and collections

`finishAggregatePull` uses a monotone IO callback (`Result<Option<Value>, string>`) for MODE,
string concatenation, and array and JSON-array collection.

- It checks the declared population and the final end of input, and keeps IO errors.
- It validates the MODE order, and keeps the first-mode tie rule and the typed representative.
- The caller gives the first retained separator or the MODE direction, and applies the canonical
  NULL, filter and DISTINCT rules before the pull.
- `finishAggregatePairs` separately uses a monotone callback of key and value pairs, for the JSON object aliases.

MODE also exposes `beginAggregateMode`, `stepAggregateMode` and `finishAggregateMode` for finite polls.

- One fold keeps its best and previous typed values, and checked run and row counters. It keeps
  the first sorted mode when populations are equal.
- NULL, input limits, decreasing order and exhausted counters are refused before mutation.
  Immutable poll versions stay valid.
- Ordinary MODE and the pull wrapper share this kernel and finisher.
- The MODE fold also has a bounded continuation codec. It binds a new trusted direction and allowances, and keeps the
  exact typed best and previous values. It uses the canonical admitted Value decoder. No input replay or population
  array is encoded.

String, array and JSON collection exposes `beginAggregateCollection`, `stepAggregateCollection`,
`stepAggregateCollectionPair` and `finishAggregateCollection`, through the same canonical writer.

- A cursor can pause between scalar or key and value inputs. It does not have to use a full hot
  group during finalization.
- Each step renders under the scalar scratch credit. Then it checks the complete appended entry
  and the necessary closing delimiter, before it changes the output or the population.
- Refused inputs keep the earlier continuation.
- Duplicate JSON keys, NULL values, separators and dialect rendering keep their meaning.
- The configured output wire limit stays mandatory. The resident mode still requires an
  equivalently large retained buffer.
- `aggregateCollectionOutputCredit` credits overlapping output carriers and separator storage.
  Scalar rendering, source pages and other operators need their own grants in the same Resources scope.
- This source implementation adds native cases for pages, COW, refusal and closed output.

#### Collection chunks

`beginAggregateCollectionChunks` uses the same writer and renderer with one bounded pending entry.

- `nextAggregateCollectionChunk` gives raw byte fragments.
- `finishAggregateCollectionChunks` queues the closing framing.
- `collectionChunkCertificate` certifies the complete result only after it drains.
- Continuation codecs keep pending fragments and the trusted definition and context. They never
  keep output that was already emitted.
- Chunks can divide UTF-8. A receiver must keep them as bytes, and atomically publish the staged
  output with its final certificate.
- Individual scalar limits, and separately admitted render, source and page buffers, stay mandatory.

The native `diskCollectionGroupedFinite` uses the external sort by equality group and source
ordinal, and stages these fragments in `TempPages`.

- Polls charge input and chunk work. Only a complete group whose framing certificate and
  temporary arena seal both succeed exposes its typed keys and certificate.
- `readDiskCollectionPayload` returns byte pages leased separately.
  `closeDiskCollectionDelivery` requires that the certified payload is drained before the next group starts.
- This cursor keeps no complete output buffer.
- It admits one scalar or pair collector in source order, bounded column and constant arguments
  and filters, and constant or parameter separators. DISTINCT and explicit aggregate order stay checked refusals.
- The caller owns each returned page until close.
- Source, render and chunk overlap use the same Resources issuer. A temporary quota failure
  leaves the output not certified, and requires cursor cleanup.
- The logical plan that the caller owns can still keep input VALUES. Sort construction is synchronous.
- This native delivery path does not give a Batch or PG scalar reader, a durable cursor restart,
  or atomic publication of a maintained view.
- The u32 text payload and the configured output wire caps stay in force.

#### Selection, pair and render credits

`aggregateSelectionCredit(maxValueWireBytes, leaseAllowance)` preflights the conservative logical
overlap `4,096 + 12 * maxValueWireBytes`. This is for two retained scalar values and the incoming
scalar, each with four times the buffer capacity.

- This covers rank selection and MODE. The cursor owns shared leases, the cadence of finite work
  and deadlines, and output consumption.
- New independent native cases include 32 bounded pages, limits of NULL, probability and precision, descending ranks,
  coalesced endpoints, binary discrete and MODE results, overflow and COW, and refusal boundaries.

The JSON object pair aliases keep the conversion of key strings, value rendering and the input
order of duplicate keys.

- No object map and no key or value arrays of the population size are made.
- These aliases do not give the duplicate-key normalization or NULL-key compatibility of PostgreSQL JSONB.

Resident collection finalizers share the transient object and array writers and the canonical
SQL and JSON renderers. There is no second aggregate state or sort.

- Result limits count the exact canonical Value wire bytes, and refuse an overflow without truncation.
- `aggregateRenderCredit` preflights a conservative credit for each scalar before rendering. This
  includes worst-case JSON escapes, vector JSON shells and overlapping render buffers.
- Input, separator, output and decode buffers, container capacity and the final output conversion
  need concurrent leases in the shared Resources owner of the caller.
- MODE can keep two scalar references. Percentile interpolation can keep two rank values.
- Credit is not measured allocation or RSS.
- The result still occupies a bounded resident Value. A stream of a result to a client, or a spill
  of its output, is a separate physical sink contract.

### Compact maintained aggregates


The function owner separates transitions that the executor attests from its one arrangement of
full-row membership.

- Ordinary COUNT keeps only counters, and finishes without an enumeration of rows.
- COUNT DISTINCT keeps canonical equality keys with counted multiplicities in the Ordered index.
  A delete removes a key only when its last copy goes. It is exact, not a sketch.
- The historical spelling `approx_count_distinct` uses the same exact compact strategy.
  - Its aggregate name stays part of the checkpoint definition.
  - Intrinsic distinct normalization applies also without an explicit SQL DISTINCT modifier.
  - NULL is excluded. An empty group returns integer zero. The last counted copy removes a
    canonical equality key.
  - It is not an approximate algorithm. It gives no sketch error limit and no measured capacity claim.

#### State records

- Native compact APIs accept and return `CompactAcc` (`count`, `state`, `inputError`), not the
  ordinary `Acc` record.
- Retained signed APIs use `SignedAcc` (`state`, `inputError`), with the counted membership and
  the bounded replay contract.
- Neither retained strategy stores unrelated ordinary numerical or model fields. Ordinary `Acc`
  no longer contains either state.
- Signed callers read `.state` in place of `.counted.unwrap()`. Compact callers read `.state` in
  place of `.compact.unwrap()`. This is a change of the native source API. Checkpoint versions and
  their encoded meanings stay the same.
- Signed reorder and merge refuse earlier errors before mutation. This storage separation does not
  make replay local to a delta.
- Optional counted and numeric strategy records are in one immutable Vec cell only when admitted.
  Ordinary COUNT holds neither strategy payload, and reserves no inline Ordered or AVL record.
- Strategy changes keep earlier versions immutable through the same Vec ownership rules.
- This layout specialization alone does not give allocator byte costs or an RSS improvement.

#### Replay

A transient `AggregateReplay` fold can use the single attested group membership of the executor.
Then each fallback aggregate does not keep argument tuples and identity and ordinal indexes.

- `beginAggregateReplay`, `stepAggregateReplay` and `finishAggregateReplay` use the ordinary
  accumulator and finisher. The standalone `finishSigned` uses the same fold.
- Copies stay bounded replay work. Ordered members must arrive in strictly increasing declared unique order.
- DISTINCT keys and typed-tie checks stay transient fold state.
- Shared membership admission excludes the MIN/MAX typed-representative fallback, collected
  aggregates, and histogram merges with an order-sensitive optional floating sum. It requires
  deterministic arguments and filters from the executor.
- Logical input and kernel-row counters and charged wire work are not allocator, RSS or latency measurements.

#### SUM and AVG

SUM and AVG keep only a numeric scalar, the multiplicity, the identity, and a stable binary SQL
order key for each contribution. They use an AVL tree of exact totals and relative prefix envelopes.

- Updates and key moves change a logarithmic number of summaries, in fixed pages of 64 nodes.
  Deleted slots are used again.
- Staged values keep earlier snapshots, because changed node pages are copied.
- Numeric pages and the ordered identity index share the persistent 32-way `PageDirectory` of
  the SDK. Thus a staged page change copies bounded directory vectors along one radix path, and
  not every unrelated page pointer.
- Ordered node slots and free IDs use the same owner. Pop releases empty directory tails, and
  keeps immutable earlier versions.
- This source structure removes the flat directory copy.

Finish:

- Safe integer and decimal results finish from the root summary. Integer prefix overflow follows
  the batch aggregate contract.
- A decimal summary finish requires a proof that no canonical 40-digit operation loses scale. If
  not, the ordered scalar fold is used.
- The FLOAT, decimal rounding and integer overflow fallbacks charge their count of surviving
  scalar replays explicitly through `compactFinishWork`.
- There is no reassociation, and no claim of floating retraction in constant time.
- DISTINCT SUM and AVG, and arguments that are not numeric, are outside this compact strategy.
  They require an explicit alternative admission.
- The executor gives unique binary order keys that agree with the comparator. The order of a
  physical scan or commit is not a substitute.

Persistence:

- Header and per-key cell APIs let one transaction persist only the changed compact cells, with
  state, output and cursor metadata.
- Recovery streams bounded cells and checks totals. The full `encodeCompact` is a snapshot operation.
- Byte budgets give the canonical serialized payload, not RSS. Optional tree state leaves ordinary
  batch accumulators without allocations for each row.
- Transition and summary-update counters give executed work, not accepted benchmark speedups.

The native finisher is `finishCompact(acc, cx, direct)`. Callers give the explicitly evaluated
direct arguments of the group output, or `[]` when there are none. There is one result owner. No
overload, default parameter or alternative percentile finisher hides these inputs.

`compactPageWork` reports the unique touched node pages, the newly made pages, the exact added retained wire bytes, the
logical directory nodes written, and the earlier slots in their bounded directory vectors. Directory work includes the
ordered identity index. Wire growth is not an allocator or RSS counter.

### Compact percentiles

Compact median, percentile_cont and percentile_disc over enforced INT values keep one counted
canonical key for each distinct value in the Ordered B+tree.

- Optional summaries of weights that are not negative support rank selection weighted by
  multiplicity in logarithmic time, in the same page pool. Ordinary maps keep empty summary
  vectors, and keep their signed payload meanings.
- Insert, replacement, delete and restore update population summaries through the same tree owner.
- Checks of population overflow and negative weights occur before mutation. Earlier versions stay immutable.
- This stores exact retained counts. It is not an approximate sketch, and not reconstructed samples.
- The ordinary and compact finishers share probability validation, NULL and empty handling, DESC
  direction and interpolation, which includes the finite opposite-sign rule. Their probability
  multiplication stays rounded floating arithmetic, not exact rational arithmetic.
- `compactPopulationLimit` owns the admitted configured limit of 2^53-1 for this strategy. Larger
  configured populations, and DISTINCT, dynamic or other-family inputs, keep signed admission or
  get an explicit refusal. Weighted Ordered itself checks exact i64 populations up to i64 MAX.
- GIZDB-COMPACT-RANK-1 and the scalar cell wire owner persist counted keys. Recovery builds the
  weight summaries again while it streams bounded cells.
- Finish reads a maximum of two weighted ranks, and expands no duplicate arguments.
- Optional descriptor and node storage and logical path counters are not proof of allocation, RSS or latency.

### Compact MODE

Compact MODE over enforced stable INT values uses the same counted scalar keys and weighted
Ordered tree as compact percentiles.

- Each weighted child summary also records its largest multiplicity. `atPeak` selects the first
  or last key at that multiplicity, without a scan of keys or an expansion of duplicate samples.
- This keeps the smallest-value tie rule of the ordinary MODE finisher, and the largest-value tie rule under DESC.
- NULL and empty behavior do not change. Counts stay exact up to i64 MAX. No floating rank calculation is used.
- GIZDB-COMPACT-MODE-1 and the scalar cells persist one key and count for each distinct value.
  Streamed recovery builds both summaries again.
- Proof of the stable type is the responsibility of the query owner. DISTINCT, mixed or unproved
  input families keep the signed strategy.
- Weighted inner summaries now contain two i64 fields. Ordinary maps still allocate none.

### Ordered reservation

Ordered owns a conservative `reservation(includeLeaf)` for one version. Consumers do not
duplicate guesses of tree fanout or node size.

- Touched node writes keep separate credit for leaf and separator key buffers. Reservation reads do not scan the index.
- `includeLeaf=false` omits only the leaf payload that the canonical bag owner of a caller
  already charged. All separator references stay charged.
- Structural credit includes bounded node vectors, optional population and peak slots, pool
  buffers and generic directory paths.
- Checked saturating arithmetic returns i64 MAX on impossible accounting. This prevents an
  admission after wraparound.
- This is a conservative logical resource reservation. It is not measured allocator or RSS bytes,
  allocator overhead, or totals for historical versions that are kept at the same time.

### Bit aggregates

Ordinary bit_and and bit_or keep exact AND and OR masks in the scalar fields of the accumulator.
They share one step kind and mask merges in constant work.

- Each argument is converted one time through the canonical integer conversion. No collecting vector is stored.
- Its NULL result finisher also serves compact bit populations.
- Signed replay folds each retained argument one time, whatever its duplicate multiplicity is.
  It keeps the non-NULL count.
- Sliding bit windows can use the merge tree after the query owner admits this scalar kernel.
  Until that predicate is integrated, they still fold each frame explicitly.

Native compact bit_and and bit_or keep 64 exact one-bit population counters in an optional
bounded payload. Other strategies do not have it.

- It keeps no argument values and no value index.
- Each transition updates a maximum of 64 populations. The finish reads those counters, and does not replay rows.
- Signed conversion uses the numeric owner, which includes the sign bit. Empty and all-NULL groups return NULL.
- DISTINCT does not change a bitwise result. The single membership arrangement of the executor
  validates duplicate and old-image multiplicities.
- Only enforced stable INT sources admit this SQL strategy. Other inputs keep signed replay.
- GIZDB-COMPACT-BITS-1 records the rows, the non-NULL count and all 64 populations in a bounded
  CRC header with zero cells. Wire accounting includes the full payload before it is allocated.
- A zero `compactFinishWork` means no replay of surviving rows. It does not mean zero counter
  work, and it is not an allocator, RSS or latency measurement.

### Boolean aggregates

Native compact bool_and, every and bool_or use exact counters of the true, non-NULL and total
populations, without an index or a retained scalar payload.

- `CompactAcc.count` is the true population for this strategy. `state.nonNull` and `state.rows`
  keep their ordinary population meanings.
- The shared Boolean result finisher also serves batch and signed replay.
- Empty and all-NULL inputs give NULL. A delete of the last true, false or non-NULL copy keeps the SQL meaning.
- DISTINCT has the same Boolean result. The canonical source membership validates multiplicity
  one time in the executor.
- `Func.compactStrategyForType` admits this strategy only for enforced stable BOOLEAN input.
  Dynamic SQL inputs keep the signed strategy.
- Native transitions use the canonical truth owner. They refuse binary arguments, and validate
  true, false and NULL underflows before mutation.
- GIZDB-COMPACT-BOOL-1 records all three populations in one bounded CRC frame. The header API
  uses that frame with zero cells. No metadata guess discards non-NULL counts.
- Constant population arithmetic does not give an end-to-end latency claim or a product parity claim.

### Compact MIN and MAX

Native compact MIN and MAX keep one counted scalar payload for each distinct raw value in the
Ordered index, after the executor validates the source membership one time.

- A delete of the last copy removes that scalar. Finalization seeks the first or last surviving
  entry, and does not replay rows.
- NULL contributes only to population accounting. Empty and all-NULL results stay NULL.
- Homogeneous families are admitted: integer, float, decimal, binary text, bool, BLOB, and
  temporal values of the same unit.
- These are refused explicitly, and need the signed strategy with a suitable order contract:
  mixed coercive families, vectors, DATE microseconds that cannot be represented, and equal values
  with different raw typed representatives (decimal scale, signed zero or NaN payload). This keeps
  the first typed representative of cold MIN/MAX, and does not choose an arbitrary byte tie.
- A nonbinary collation requires separate admission evidence.

SQL eligibility:

- `Func.compactStrategyForType` is the SQL eligibility owner over the only `compactStrategy` declaration.
- SQL first uses compact extrema only for enforced stable INT, BOOL, binary TEXT and BLOB
  expression types. FLOAT, DECIMAL, temporal and dynamically typed input keep the signed strategy
  until preflight and fallback keep evaluated arguments and typed representatives.
- A native transition refusal does not permit the refusal of a SQL query that is now valid.
- The caller must prove the stable type contract before it selects the compact strategy.

Persistence:

- The header and cell streaming APIs carry raw byte keys. Extrema use cell kind 2 and full
  GIZDB-COMPACT-EXTREMA-1 snapshots.
- Canonical `orderBytes` framing and Value wire encoding own the typed key. Recovery computes it
  again, and verifies the family, multiplicities, totals, CRC and limits.
- A conservative key scratch limit and exact retained wire credit are different from allocator or RSS evidence.
- Directory counters give bounded path copy, not latency or physical memory.
- These contracts do not give parity with R or with database products.

### Typed metric analytics

The statistical function owner exports `MetricCounter` and `MetricHistogram`, with validation,
conversion between cumulative and delta, counter rates, spatial histogram merge, and
nearest-rank quantile containment. SQL uses the same implementations through the [metric functions](../reference/sql.md#metrics).

#### Carriers

- The versioned JSON text carriers use `giz.metric.counter.v1` and `giz.metric.histogram.v1`.
- Timestamps, counts and i64 counter values are canonical decimal strings. Thus values above 2^53 stay exact.
- The counter `number_type` is `i64` or `f64`. Floating values must be finite.
- Decoding validates the carrier through the SDK pull reader. It keeps only the final value span
  of each declared field, so the last duplicate key wins.
- Overwritten and unknown nested values are syntax-checked without a JSON tree.
- Native array widths are checked before growth. Escaped strings get raw-span and decoded-byte
  checks before admission.
- The carrier has a limit of 1 MiB. The limits of 4,096 bounds, 4,097 counts, 4,096 sources and
  65,536 total identity bytes stay.

#### Decode and overlap credits

`metricDecodeScratchBytes(value, leaseAllowance)` preflights a conservative logical credit of
`65,536 + 64 * carrierBytes`. This is for scanner and field metadata, input and slice, and the
overlap of bounded decoded vectors and strings.

- It validates the carrier type and size, and the allowance, before the parse.
- It is not an allocator or RSS measurement.
- Disk execution must get this shared-scope grant before the decode. It must separately cover
  canonical native validation, the resident point, merge operands, and simultaneous encoded input and output.
- Disk histogram admission stays refused until those overlapping grants are attached. The decoder
  alone does not prove that path.

`aggregateKernelOverlapBytes(name, star, arity, stateWireAllowance, inputCarrierBytes,
outputWireAllowance, leaseAllowance)` owns a combined histogram grant for the shared query lease of an operator.

- It covers pooled, incoming and merged native points at full native caps, typed carrier decode,
  validation, merge and renderer scratch, and simultaneous state and result wire buffers.
- Full caps are necessary because native growth and refused rendering can occur before the
  configured wire and output check. A very small output limit never makes that credit zero.
- Larger configured carrier and frame limits saturate at the native carrier cap before the arithmetic.
- The maximum credit is 107,848,052 logical bytes. The helper does not prove allocator, RSS or disk-path use.
- The operator must enforce its input carrier limit, stage only successful state, keep the grant
  through rendering and typed output, and release it through the resource lifecycle.
- The native parser, merge and finisher stay canonical.

`aggregateKernelTransitionBytes(name, star, arity, acc, inputCarrierBytes, outputWireAllowance,
leaseAllowance)` also derives an adaptive overlap from the actual retained shape of buckets,
sources and identities, and from the admitted raw input carrier limit.

- It checks retained structural sizes without an allocation of validation maps or hashes.
- Raw carrier bytes give a conservative limit for incoming decoded identities, array geometry,
  and also invalid empty-source entries. Native caps limit merged growth before any final size check.
- The same native, decoder, renderer and sealed-frame size owners charge the old, incoming and merged overlap.
- Refused rendering stays charged, also with an output allowance of one byte.
- The unchanged full-cap grant stays available as a conservative alternative.
- A new state with an input carrier of 300 bytes and a result allowance of 65,536 bytes reserves
  701,848 logical bytes, not the global native maximum.
- This is an arithmetic admission contract. It is not an allocation, RSS or latency measurement.
- The executor must resize its shared lease before decode and step. It keeps unrelated source,
  sort and aggregate grants, and keeps the output overlap until consumption.
- A zero incoming carrier can preflight finalization. It does not discard the geometry of the retained point.

#### Counters

- `series` declares the identity of resource, scope, name and attributes. `unit` is part of the identity.
- `start` and `time` are nanoseconds since the Unix epoch, not negative, with a limit of signed i64.
- `temporality` is `delta` or `cumulative`. Counters also declare the boolean `monotonic`.
- Counter differences keep i64 arithmetic. They refuse overflow, decreases without a monotonic
  reset, changes of identity, unit or representation, and overlapping reset epochs.
- A cumulative point with an unknown start (`start=0`), or an initial reset with zero duration and
  an unknown start (`start=time`), gives no initial delta or rate. Keep that original point as the
  baseline of the next conversion.
- Resets with a known start give the total of the new epoch.
- Delta accumulation requires contiguous known intervals. Gaps, overlaps and missing starts are explicit errors.
- Counter rates are floating values for each second, in the unit of the counter.

#### Histograms

- `bounds` are finite numbers that strictly increase. `counts` has bounds+1 exact i64 entries,
  not negative, whose checked total is `count`.
- Upper bounds are inclusive. Lower bounds are exclusive. Empty bounds with one count are one unbounded bucket.
- `sum` is optional, finite and not negative. It is zero for an empty population. Omit it for
  signed observations. A missing sum stays missing during conversion or merge.
- Histogram conversion refuses changed geometry and decreases without a mark.
- A spatial merge requires identical intervals, units, geometry and temporality, and disjoint declared `sources`.
- Pooled results keep the union of the sources and a canonical population identity. Thus an attempt
  to merge them again with a constituent is found.
- Producer provenance is an assertion of the caller. It is not authentication.
- The aggregate stores one histogram in the accumulator, propagates validation errors, skips SQL
  NULL and returns NULL when there is no input.

Merge order:

- Bucket populations and the provenance union are exact.
- An optional f64 sum follows pairwise IEEE pooling, and can change with the order of arrival or
  with partition groups. A native vector merge sorts the given series, but pooled summaries cannot
  restore an original order of summation.
- Thus the canonical aggregate property marks a histogram merge as order sensitive. A continuous
  signed route requires its unique total-order admission contract, although bucket populations are exact.
- This does not permit the refused disk histogram route. Shared-membership replay keeps the
  retained route of this aggregate. No scalar sample values or extra sum state are reconstructed
  to hide that numerical difference.

Quantiles:

- Quantiles return JSON bucket containment: `lower`, `upper`, inclusivity, and `exact=false`.
- They use an exact rank calculation with binary mantissas and integers, also above 2^53.
- A null endpoint means an unbounded tail. Empty populations return SQL NULL.
- Bucket counts cannot identify an exact sample percentile. Thus no interpolated sample or finite
  tail error is invented.

Limits and scope:

- Admission caps are 4096 boundaries, 4096 population sources, 64 KiB of aggregate identity bytes
  and a JSON carrier of 1 MiB. These are limits of encoded input. They are not an RSS guarantee.
- Optional min and max, exemplars, flags, exponential histograms and OTLP transport are not part
  of this carrier. JSON fields that are not supported fail explicitly.
- No parity with Datadog or with complete OTel is claimed.
- Interval and bucket semantics follow the
  [OpenTelemetry metric data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/)
  and the [OTLP metric schema](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto).

### Time buckets

`time_bucket(interval, ts [, origin])` rounds a time down to buckets from 2000-01-03 (months
from 2000-01-01), as TimescaleDB does.

### Column kernels

Column work runs in SIMD kernels (`db/core/kernels.giz`, 8 lanes of 64 bits):

- comparisons of a column with a constant or with a column;
- `+ - *`, with the overflow check of the scalar path;
- AND and OR in three-valued logic, and IS NULL;
- selection vectors;
- count, sum, min and max for one group (NaN is the greatest, as PostgreSQL orders it);
- decimal columns of one scale against a constant, and decimal `+ - *` on their coefficients
  (exact. Past 64 bits, the row path rounds as NUMERIC does);
- EXTRACT one time for each distinct date;
- vector distances to a constant vector;
- segment packing (`bench/db` olap).

The machine decides, never a constant written for one host (`gizlang:machine`, `docs/internals/compute.md`):

- A compiled kernel unit is built for the x86-64 level of the host (`-march=x86-64-v4` where AVX-512 is
  usable), and a process skips a cached unit above its level (`GIZLANG_CPU` caps both).
- Cache-sized tables take their size from the core's share of the last-level cache.
- `pg_stat_gizdb_machine` shows what GizDB sees, one `name`, `value` row a fact: `arch`, `level`,
  `features`, `cpus`, `threads`, `cache_groups`, `nodes` and `lN_share` (bytes of cache level N a core may count on).

Source batch 2026-10-11:

- Integer SIMD sums prove from the sign bounds of each value that every ordered prefix fits.
  Values outside that proof use the ordered row fold. Thus lane cancellation cannot hide an integer overflow.
- Integer aggregate states keep relative prefix minimums and maximums and an exact total. Only
  pieces that overflow allocate wide decimals.
- Ordered piece merges translate the full prefix envelope. Thus an intermediate overflow cannot
  disappear through cancellation, and does not occur only because a later piece is aggregated alone.

Performance evidence uses `bench/db/roofline/run.giz` and `bench/db/tpch/run.giz`.

- They check complete nullable result cells, exact reference integer, decimal and text types, and
  finite floating tolerances.
- They record the first execution, and the median and range of five later executions.
- The Roofline denominator is a logical model of a full scan of the named columns. It is not an
  unconditional physical lower limit. Pruning, indexes, cache residency, intermediate
  materialization and actual parallelism need separate plan and buffer evidence.
- Historical measurements and assumptions stay in `bench/db/README.md`.

### Session

`session.giz` runs autocommit, or BEGIN, COMMIT, ROLLBACK and savepoints, over the transactions of
the engine, at each isolation level (§3).

- Every lookup, index probe and scan is recorded as a read for the commit to validate.
- A btree index answers equality on each prefix of its columns. A read through a prefix conflicts
  only with writes of that prefix. An index on `(chat, seq)` serves `chat = $1`.
- Statements: INSERT (defaults, identities, ON CONFLICT, OR REPLACE and OR IGNORE, RETURNING),
  UPDATE and DELETE (with FROM), CREATE TABLE [AS], indexes, views, sequences, ALTER TABLE ADD and
  RENAME COLUMN, and DROP.
- `nextval('sequence')` and `currval('sequence')` with a literal name use the same sequence owner
  as `sequence.NEXTVAL` and `CURRVAL`. Supported calls in the SELECT list and in VALUES share row numbers.

### Numbers

`db/decimal.giz` and `types.giz`: DECIMAL, NUMERIC, NUMBER(p,s) and MONEY are exact decimals.
They have a 128-bit coefficient and a scale, and use one i64 while the value fits. Thus
0.1 + 0.2 = 0.3, and a sum of cents is cents.

- A column keeps its declared scale (rounded half away from zero) and precision (22003 past it).
  A column without a scale keeps the scale of each value.
- Addition and multiplication are exact. Division takes the result scale of PostgreSQL. Past 38
  digits, a result gives up fraction digits, and then fails (22003).
- Columns whose coefficients fit 64 bits use the batch kernels. Wider columns use the exact row path.
- Integers, floats and decimals order exactly against each other (one key encoding).
- As in PostgreSQL, a decimal with a float compares and computes as a float. CASE and COALESCE
  arms of mixed number types are cast to the type that they resolve to.

| Dialect | Integer result past 64 bits |
| --- | --- |
| PostgreSQL, MySQL | Error (22003) |
| Oracle | An exact NUMBER |
| SQLite | A float |

- Floats read NaN, Infinity and -Infinity.
- SQLite keeps its affinity (NUMERIC holds reals).
- PostgreSQL text-to-integer casts require a signed decimal integer spelling. They refuse
  fractional, exponent and junk text. Numeric-to-integer casts keep their rounding rule.

### Binary values

JSON stays checked text. Native BLOB and BYTEA use `Value.blob(bytes)` and a packed column arena
with row offsets and a separate NULL bitmap.

- Empty bytes are different from NULL. Equality and indexes keep raw bytes and unsigned byte order.
- The only tuple-key encoder returns opaque bytes. Native indexes, transaction write tracking, SQL
  joins, groups, sets and subqueries, and DISTINCT state use it.
- Identifiers stay checked UTF-8 text. Arbitrary keys never go through a string conversion.
- The WAL, schema and segment readers keep their previous tags. Binary adds Value tag 11, Type tag 9 and Column tag 9.
- PostgreSQL OID 17 supports exact binary parameters and results, and hexadecimal or legacy
  escaped text input. Text output uses `\x` plus lowercase hex.
- Numeric and implicit text operators refuse binary inputs. Explicit text casts render the canonical representation.
- Parquet byte arrays without an annotation keep bytes. STRING, ENUM and JSON annotations require UTF-8.

#### Native binary function admission

- BLOB and bytea stay raw through COUNT, exact DISTINCT, min and max, arbitrary-value and discrete
  order-statistic accumulation.
- Byte `length` and `octet_length`, explicit hexadecimal display, SQLite `quote` and `typeof`,
  PostgreSQL `pg_typeof` and `sha256(bytea)` operate on the original bytes, also with NUL and invalid UTF-8.
- Binary SHA256 returns binary digest bytes. Inherited text hashing still returns hexadecimal text.
- Explicit PostgreSQL `to_json(bytea)` renders the canonical bytea text in a JSON string. The parse
  of a model or JSON document from arbitrary binary bytes is not supported.
- Numeric, implicit text and unsupported collected-value functions refuse binary arguments.
- Ordinary and inverse-window numeric accumulators propagate input errors, also from a partition
  with a count of zero.
- Binary string aggregation, arrays and JSON collections, and continuous percentiles stay explicitly not supported.
- References for byte operations: [PostgreSQL binary strings](https://www.postgresql.org/docs/current/functions-binarystring.html)
  and [SQLite core functions](https://www.sqlite.org/lang_corefunc.html).

#### Kafka binary rows

Kafka payload and header rows use topic metadata version 2 and native binary columns. Text rows of
version 1 require the explicit trusted
`Store.migrateLegacyIn(engine, tx, tenant, topic, maxRows, maxBytes, backing)`.

- It requires a clean caller transaction, a complete contiguous offset cut, the exact schema, the
  canonical legacy encoding, and positive independent credits of physical rows and bytes.
- Historical delta and segment versions are charged before scratch for the visibility scan is allocated.
- It stages no mutation when preflight refuses. It fences concurrent writes at every isolation
  level. It refuses other open snapshots at staging and commit, through the common quiescent-rebuild fence of the Engine.
- Converted rows and the unchanged next offset publish with the replacement schema and one Catalog
  revision, in the transaction of the caller. The caller commits and syncs through the durability owner.
- Larger cuts outside the selected credits are refused. There is no automatic conversion, partial
  publication or silent fallback.

### Dialects

In `dialect.giz`, the dialect of a session is a value of the rules where PostgreSQL, Oracle, SQL
Server, SQLite and MySQL differ in meaning. Examples:

- the empty string of Oracle as NULL, and its exact integer division;
- the type affinity of SQLite;
- LIKE case, NULL order and boolean text;
- DATE with time;
- what an error inside BEGIN does;
- the default isolation.

Every module reads the rule. Thus a dialect is a new value, not edits across the engine.

- A client selects it with `SET sql_dialect`, or `options=-c sql_dialect=oracle` at startup.
- Oracle clients connect over TNS on the same port as PostgreSQL clients (`db/ora/server.giz`): the first bytes tell the protocol. The server does O5LOGON against the role's Oracle verifier (set with its password) or the server password, then relays each statement into an `oracle` session. A session prepares each statement text once (up to 256), and every cursor of that text runs it with one Bind and Execute; a forwarding loop then answers a read or write by key as it does a driver's named statement. It holds a query's rows and serves fetches from the hold. A column described as text is fetched as text, and every other column in binary, so a value is never read as another kind than the one it is written as. `tests/db/ora-server.test.giz` logs on with the Gizlang client (`db/ora/client.giz`).
  - The server speaks the 12c and later TTC. JDBC thin 23, python-oracledb thin, node-oracledb thin and go-ora (field version 11) log on and run the same checks, and the server derives each client's wire form (token numbers, chunked strings, the end-of-call and ECID fields of a summary) from the field version the client announces. The logon answer carries the limits a client sizes its caches from (`AUTH_MAX_OPEN_CURSORS`), and `NLS_SESSION_PARAMETERS`, `NLS_DATABASE_PARAMETERS` and `V$NLS_PARAMETERS` answer the NLS queries drivers run at start. A statement can bind IN, OUT and IN OUT values, `RETURNING ... INTO`, and a REF CURSOR out; the DBMS_OUTPUT buffer is read with `GET_LINE`; DatabaseMetaData and `DESCRIBE` (function 119) are answered from the dictionary views.
  - A CLOB or BLOB column is described as a LOB (a `LOB_MOD` type modifier marks it between the SQL layer and the wire) and its value is a locator of the session. The server holds the LOBs a session fetched or created in memory (64 MiB, oldest data evicted first) and serves the LOB operations (length, read, write, trim, temporary LOBs, chunk size) on them. A select with LOB columns is described first, and the client then re-executes it with defines and a LOB prefetch, as Oracle does. A write through a locator of a stored LOB is not saved to the table; assign the value with an UPDATE.
  - The text of the ORA-nnnnn errors follows Oracle: ORA-00942 names `"SCHEMA"."TABLE"`, an unknown function is ORA-00904.
- The text formats of the PG protocol stay those of PostgreSQL.
- One grammar accepts the spelling of every dialect (`parse.giz`).
- A PostgreSQL session also takes the MySQL and StarRocks functions and operators that analytics
  queries use, with StarRocks' answers and a type a driver reads as a number or a time:
  `DATE_FORMAT` and `STR_TO_DATE` with `%` elements, `DATE_ADD`, `DATE_SUB`, `ADDDATE`, `SUBDATE`
  and `INTERVAL n UNIT` (a date or a timestamp without a zone gives a timestamp), `WEEKDAY`,
  `UNIX_TIMESTAMP` (a wall time is the session zone's), `UTC_TIMESTAMP`, `DIV`, `MOD`, `PMOD`,
  `CRC32`, one-argument `ISNULL`, `SUBSTRING_INDEX`, `FIELD`, `GET_JSON_STRING`, `GET_JSON_INT`,
  `GET_JSON_DOUBLE`, `PARSE_JSON`, and the aggregates `MAX_BY`, `MIN_BY` (a NULL key is skipped),
  `COUNT_IF` and `PERCENTILE_APPROX` (exact, as `percentile_cont`).
  `tests/db/starrocks-funcs.test.giz` holds StarRocks 4.1's answers. `CAST(x AS CHAR)` stays
  PostgreSQL's `char(1)`; write `AS STRING` or `AS TEXT`.

### Routines

`routine.giz`, `proc.giz`, `plexec.giz` and `trigger.giz` give CREATE FUNCTION, PROCEDURE and TRIGGER, CALL, DO
and anonymous blocks, in these forms:

- PostgreSQL (SQL, PL/pgSQL);
- Oracle (PL/SQL, :NEW and :OLD, DBMS_OUTPUT, RAISE_APPLICATION_ERROR);
- SQLite.

Execution:

- A function in a query is not interpreted for each row. A SQL function, or a PL function of
  declarations, assignments, IF, CASE, SELECT INTO and RETURN, becomes part of the query as an
  expression or a subquery (Froid, VLDB 2018). Thus the optimizer and the column kernels see through it.
- Any other function runs one time for each distinct argument tuple of the rows that the FROM and
  WHERE of the statement give. The plan reads its results by argument.
- Procedures, triggers and those functions run in the interpreter of the session. Each embedded
  statement is parsed and planned one time for each connection.
- Exception blocks undo their writes in PL/pgSQL, and keep them in PL/SQL, as each engine does. A
  failed statement undoes only itself.
- Triggers: BEFORE and AFTER, ROW and STATEMENT, UPDATE OF, WHEN, and a BEFORE trigger that
  changes or skips the row.
- PL/SQL follows Oracle Database 23 (the answers in `tests/db/oracle/*.out` come from it, `known.txt` lists the
  cases that differ): packages (a spec and a body, state kept for the session, an initialization block, overloads,
  calls without parentheses), collections (nested tables, VARRAYs, associative arrays, records, `%ROWTYPE`, their
  constructors and methods, `BULK COLLECT`, `FORALL`), `PRAGMA EXCEPTION_INIT`, `PRAGMA AUTONOMOUS_TRANSACTION`
  (the routine runs in a transaction of its own, which its COMMIT ends), `GOTO` and labels, and ORA-04091 for a row
  trigger that reads or changes its own table. A collection is a package variable, a parameter and a result (it crosses
  a call as one packed text, `db/sql/types/collpack.giz`); `CREATE TYPE t AS TABLE OF e` and `VARRAY(n) OF e` make a type
  of the schema; `TABLE(c)` of a variable or a function result is a FROM item with the column `COLUMN_VALUE`;
  `x MEMBER OF c`, subprograms nested in a block, `RETURNING INTO`, `EXECUTE IMMEDIATE ... INTO/USING` and
  `FOR UPDATE` with `WHERE CURRENT OF` work. Errors show the ORA-nnnnn texts of Oracle. A call qualified with a
  package name (`pkg.total(x)`) is the package routine, never a built-in aggregate of the same name.
  - Object types: `CREATE TYPE t AS OBJECT (attributes, MEMBER and STATIC functions and procedures)` with
    `CREATE TYPE BODY`, `UNDER` for a subtype, the constructor `t(a, b)`, columns of an object type, `x.attr`,
    `col.attr.method()`, `TREAT`, and `DROP TYPE ... FORCE`. In PL/SQL an object variable has its attributes and
    calls its members; a procedure member writes SELF back. `USER_TYPES`, `USER_TYPE_ATTRS` and `USER_TYPE_METHODS`
    list them. A member call runs the method of the declared type (no dynamic dispatch to a subtype's override).
  - `PIPELINED` functions with `PIPE ROW` (read with `TABLE(f(x))`), `WITH FUNCTION` in a query, `INSERT ALL`
    and `INSERT FIRST`, an update of a join view, and ORA-06531 for a method on an uninitialized collection.
  - Not supported: collections of records in `TABLE()`, collection OUT parameters, dynamic dispatch of an overridden member.

**Not supported:** calls for each row of functions that are not inlined inside subqueries, and INSTEAD OF.

### PostgreSQL protocol

In `db/pg/`, one event loop owns the engine. Each connection has a session.

- With `--threads N` (by default one per CPU the process may use), N more loops share the port, each on its own
  thread. A read-only statement outside a transaction runs there, against a snapshot that the loop of
  the engine publishes, when that snapshot holds every commit that was already confirmed to a client.
  All other work goes to the loop of the engine (`bench/db`).
- A loop also mirrors a transaction that the engine's loop holds (`db/server/forward.giz`). It starts at a plain BEGIN, or
  at a unit that has BEGIN before its first statement, as pgjdbc and the Oracle bridge send them (`beginUnit`).
  - The loop runs each later statement on its snapshot and sends its writes and reads as one write set. The engine
    checks the reads against its own state and takes the set (`pg_stat_gizdb.shipped_adopted`), or runs the unit
    itself.
  - A unit that the engine runs (FOR UPDATE, a named Parse, a failed set) carries a mark (SHIP_MARK). The engine
    answers it with the writes that the unit added (`shipped_acked`), so the mirror continues.
  - A point read by the whole primary key, of one or more columns, answers on the loop when its snapshot is the
    newest and the transaction did not write that row (`local_reads`).
  - A point UPDATE, INSERT or DELETE of a named statement by its whole key (integer, text or numeric columns) is a
    put or a delete that the loop makes without the statement path (`shipped_rows`), unless the transaction already
    wrote that row. An Execute row limit does not stop it (pgjdbc's executeUpdate sends 1): a write with no RETURNING
    runs whole, as in PostgreSQL.
  - When the loop's snapshot is the newest, it answers a write at once and sends the set after it (SHIP_AHEAD,
    `shipped_ahead`). A unit with BEGIN before its first statement answers at once too (SHIP_BEGIN). While such a
    write is not acknowledged, the reads that the loop answers go to the engine too. The engine takes the set, or
    runs the statements again and compares their answers with what the client got. A different answer makes the
    transaction fail at COMMIT with 40001, as a serialization failure does in PostgreSQL.
  - An error, the end of the transaction or a missing acknowledgement ends the mirror.
- It supports the simple and the extended query protocol (Parse, Bind, Describe, Execute, Sync),
  text and binary results, and SQLSTATE errors. psql, pgbench and `gizlang:postgres` connect unchanged.
- **Group commit.** A commit applies immediately and is written to the log. The log sync runs in a
  task of its own while the loop takes more commits, and the next sync covers all of them. A commit
  shows to other sessions once it is durable, as in PostgreSQL: a snapshot is the durable clock, so a
  read never waits for a sync, and only the reply of a client's own commit waits. A RESP session whose own commit is
  not durable yet reads the newest commit, as its writes do, and that reply waits for the sync too: a pipeline's GET
  reads the SET before it. A standby gets only
  the synced log. Thus no client sees what a crash can lose. (Under KV appendfsync everysec or no, a
  commit shows at once and RESP replies leave after the log write. As in Redis, the commits of all RESP clients of a
  loop pass, up to the point no connection is ready, go to the log in one write, and then their replies go.)
- `changes('table', after [, wait_ms [, 'column', value]])` returns the committed changes after a
  position: lsn, deleted, then the row. It gives SQLSTATE GZ001 when the position was trimmed.
  With a wait, an empty answer parks the connection until a commit writes the table or the wait ends.
- A native watch (`Access.observe`, which `Data.follow` opens) keeps only its selection: the rows of one table that a
  filter picks. Each read takes that table's changes after its position through `changes()` with the filter as a
  column, as the watch's role may read them. Thus a watch holds no result state. A position that the WAL no longer holds,
  or a watch that expired, answers `lost`, and the page reads again. An UPDATE that moves a row out of the
  filter is not reported: the journal gives the row's new image only (a deleted row is its before-image).
- **COPY** FROM STDIN and TO STDOUT (`db/sql/copy.giz`), in the text and CSV formats of PostgreSQL.
  It reads the chunks that the client sends, and keeps them as column batches of 64k rows. Each batch
  is released when the insert has read it. A load parses a batch in a part per core (CSV with quotes is cut where the
  quotes before a line end are even), and each core converts its
  part's fields into the table's columns (`Conn.bulkChunks`); the load's thread only joins them, sorts the keys and
  writes the chunk. A chunk is one key run of up to 131,072 rows (`CHUNK_ROWS`, as DuckDB's row groups of 122,880), and
  its encoded segment is one WAL frame of at most 64 MiB (`MAX_RANGE_BYTES`); a segment past it is staged as two runs.
  Runs of 11K rows (a 16 MiB frame, cut by input bytes) made each scan pay a dictionary, zones and setup per 11K rows:
  ClickBench at 10M rows ran 0.65 of the time in runs of 127K rows. A server file is refused.
- Passwords authenticate by SCRAM-SHA-256.
- **Not supported:** COPY BINARY.

### Search

Search is not a separate API. It is part of the algebra:

```sql
-- full text: rows that match, scored by BM25
SELECT id, title, score() FROM docs WHERE match(body, 'raft consensus') ORDER BY score() DESC LIMIT 10;
-- nearest neighbours under a filter
SELECT id FROM items WHERE tenant = 7 ORDER BY distance(embedding, [0.1, 0.2, ...]) LIMIT 10;
```

- `match` reads a full-text index of its column: `CREATE INDEX docs_body ON docs USING bm25(body)`.
- The inverted index answers `match`, with Block-Max WAND for top-k. It skips blocks whose best
  possible score cannot enter the current top-k.
- BM25 statistics (document count, average length, document frequencies) are summed over the
  segments and the delta of the snapshot. Thus scores do not depend on how the data is divided.
- `distance` with `ORDER BY ... LIMIT k` becomes a top-k nearest-neighbor source.
  - It uses an exact scan when the filter leaves few rows (cheaper and exact). If not, it uses the vector graph.
  - It applies the filter during the graph walk, not after. Thus a selective filter does not make the result empty.
  - Then it ranks again by exact distance.
- `<->` and `l2_distance` give the Euclidean distance. `<=>` gives the cosine distance. `<#>` gives
  the negative inner product, as in pgvector. The graph keeps squared distances internally.
- A hybrid search is one query: `match` OR `near(distance)` in `WHERE`, `ORDER BY rrf() DESC` and a `LIMIT`.
  BM25 scores and distances are not on one scale, so the ranks are fused (reciprocal rank fusion).

```sql
SELECT id, title, rrf() AS score FROM docs
WHERE tenant = 7 AND (match(body, 'raft consensus') OR near(embedding <=> $1))
ORDER BY rrf() DESC LIMIT 10;
```

  - The planner (`db/sql/hybrid.giz`, called before binding) writes it as two ranked subqueries of the table and a
    FULL JOIN on the table's key (the hidden `rowid` for a table with no key). The text side is the `match` top-k,
    the vector side the `distance` top-k; each ranks `greatest(50, LIMIT + OFFSET)` rows under the other
    conditions of `WHERE`. Each select item is computed on both sides and taken from the side that has the row;
    `rrf()` in the select list is the fused score. Equal scores keep the order of the key.
  - The fused score is `rrf(text_rank, vector_rank, k, text_weight, vector_weight)`: the sum of
    `weight / (k + rank)` over the sides that ranked the row (a NULL rank adds nothing). k is `search.rrf_k` (60),
    the weights `search.text_weight` and `search.vector_weight` (1). A negative or non-numeric setting is an
    error at the next hybrid search. `rrf` with three to five arguments is also an ordinary function.
  - A hybrid search with `GROUP BY`, `DISTINCT`, a window, a set operation, another `ORDER BY` or no `LIMIT` is
    an error that names the form.
  - `$1` is the query's embedding, or `pgml.embed('MODEL', 'raft consensus')::vector` in the query ([SQL ML §8](sql-ml.md#8-transformers)). A
    cross-encoder can then order the fused rows again (`pgml.rank`). The console's Data area writes this query
    from a form. `pgml.tune_search` measures the settings against judged queries ([SQL ML §8](sql-ml.md#8-transformers)).
- The retrieval metrics of one query compare a ranking with the relevant ids: `recall_at_k(ranked, relevant, k)`,
  `precision_at_k`, `reciprocal_rank`, `average_precision` (trec_eval's) and `ndcg_at_k(ranked, relevant, k, gains)`
  (trec_eval's ndcg_cut; `gains` is the graded relevance of each relevant id, 1 when it is omitted). The ids are two
  arrays of one type; k 0 or NULL is all of the ranking. `avg` over the queries gives MRR and MAP. They and `rrf` are
  `gizml:ml`'s, so a program that does not use the database computes the same numbers.

Policies (§1) are predicates added to every scan and search source before execution.

## 7. Sources

A source is a remote system that a query reads through one typed contract (`db/source.giz`). Its adapter (PostgreSQL
through `gizlang:postgres`, ODBC through `gizlang:odbc`, files) reports what it can do (schema, projection, filter,
group, join) and returns column pages. The plan gives the adapter what it accepts and computes the rest. The data is
as fresh as the remote system. Sources have no common transaction. An adapter never claims a snapshot or a change
feed that its system does not give. Following a remote change log (logical replication, binlog) into the local Log is
not supported.

### Files

**Built** (`db/sql/source.giz`, `db/core/parquet.giz`). With `--files DIR`, a superuser reads files
under DIR as tables, as `pg_read_server_files` of PostgreSQL permits.

- `FROM 'sales/*.parquet'` (the format comes from the extension), `read_parquet`,
  `read_csv(path, delim = '|', header = true)` and `read_json` (newline-delimited), over a glob
  (`*`, `?`, `[..]`, `**`).
- The open reads only what fixes the schema: Parquet footers, and the first MiB of a text file. It
  finds the delimiter, the header and the column types: boolean, integer, float, date, timestamp,
  or else text.
- A query reads units (a row group, a text file) one wave at a time. It decodes them on every core
  into the columns that a segment holds, and only the columns that it reads.
- Row groups whose min and max statistics exclude its conditions are skipped.

Parquet support:

| Supported | Not supported |
| --- | --- |
| Flat schemas, and a list of FLOAT or DOUBLE as a vector (a NULL or empty list is NULL; one length a column) | Other nested columns |
| UNCOMPRESSED, SNAPPY, GZIP, ZSTD and LZ4_RAW pages (COPY TO writes them uncompressed) | LZ4 (Hadoop framing), BROTLI |
| Every encoding: PLAIN, dictionary, RLE, the DELTA encodings, BYTE_STREAM_SPLIT | |
| Pages v1 and v2 | Object storage |
| Decimals to 38 digits, dates, times and timestamps (MILLIS, MICROS, NANOS, INT96) | |
| Writes: `COPY ... TO FILE (FORMAT parquet)`, vectors as LIST<FLOAT> | |

Rules:

- Each glob matches a file one time, also with repeated `**` segments.
- Execution checks the column names and types of the bound source again, also when a query gives
  them aliases. Changed CSV headers or field counts, new JSON keys, and incompatible typed values
  fail. They do not move columns or drop data.
- Missing JSON fields are NULL. An empty string in a text column stays an empty string.

Tests:

- `tests/db/sources.giz` exercises these contracts, signed 128-bit decimal pages, and malformed or truncated input.
- External plain and Snappy fixtures that DuckDB wrote (200,000 rows each, 12 flat columns) were
  also checked through the SQL API against the counts and aggregates of DuckDB. This is
  compatibility evidence for those fixtures. It is not evidence for every possible writer or encoding.

### Moving a database into GizDB

`gizdb migrate` (`db/ingest/migrate.giz`) copies a StarRocks database (through its FE HTTP SQL API) into GizDB,
verifies it with checksums, and runs the same queries on both sides to compare the results. The adapter contract is
`db/ingest/source.giz`: each adapter owns schema translation, snapshot pages, saved progress and checksums.
StarRocks is the only adapter. Changes after the copy are not followed.

```sh
gizdb migrate copy    --source starrocks://USER@HOST:8030/DB --password-file PATH --target DIR
gizdb migrate verify  --source starrocks://USER@HOST:8030/DB --password-env NAME --target DIR
gizdb migrate compare --source starrocks://USER@HOST:8030/DB --password-file PATH --target DIR --queries QUERY-DIR
gizdb migrate status  --target DIR
```

- Freeze source writes from the copy until the comparison ends. StarRocks has no global snapshot or change log.
- Give the password as a file or an environment variable name, never in the URL.
- A table without a primary key (a StarRocks DUPLICATE KEY table) is copied by offset in the order of its columns, so
  equal rows stay equal rows. `--key TABLE=COLUMN[,COLUMN]` (repeatable) declares a unique key for such a table; the
  copy is then keyed and fails at the first repeated key.
- `--tables a,b` selects tables. `--where "TABLE=PREDICATE"` (repeatable) filters one table. Copy, verification and
  checksums use the same predicate, which is validated inside a read-only SELECT. Use the same options to resume
  and to verify. A changed filter or time zone needs a new target.
- Copy and verification send one HTTP request at a time, in keyset pages. `--batch N` (1 to 1000, default 1000)
  sets the page size. `--timezone ZONE` sets the session time zone (default `+00:00`).
- A copy resumes where it stopped: rows and the cursor commit together. `--max-pages N` stops after N pages per table.
- Verification reports row counts, SHA-256 checksum sums and the first differing key per table.
- `compare` runs each `.sql` file (one read-only SELECT or WITH ... SELECT) on both sides and streams the full
  result with bounded memory; there is no row cap. `-- Unique order: COLUMNS` in the file declares an ordered result;
  other results compare as multisets.
- `--json` prints one report. Any mismatch or failure exits nonzero. Cutover is manual.
## 8. Correctness

### Oracle tests

Shared SQL queries in `tests/db/sql.giz` run against SQLite on the same data: tables of 2000 and
40000 rows with NULLs, before and after writes. The rows must agree, in order when the query
orders them.

- ASOF queries are compared with correlated SQLite queries for the nearest row.
- Regression and covariance use closed-form linear data, in serial and parallel execution.
- File schema drift, and bounds that are not supported or not valid, assert errors.

SQLite windows:

- SQLite SUM, AVG and TOTAL windows use the accumulator with compensated additions and inverse
  transitions, in the native phase order of SQLite.
- Approximation, the result storage type and the overflow history survive an empty frame. CURRENT
  ROW and zero FOLLOWING keep their different transition phases.
- Decimal windows keep the exact decimal extension of GizDB. Ordinary aggregates keep their
  prefix-overflow and statistical moment contracts.

Mixed columns:

- Temporary heterogeneous expression columns use `Column.mixed(Vec<Value>)`. Each cell keeps its
  kind through materialization, gather and concatenation. Typed kernels fall back to scalar evaluation.
- Stored segments stay typed columns coerced to the schema, with unchanged file tags. Mixed columns
  cannot be encoded as segments.
- The reference matrix for inverse windows and heterogeneous columns is connected to the DB scenario.

### Generated queries

`tests/db/gen.giz` uses the method of SQLancer.

- 10,000 typed random queries under a fixed seed agree with the rows of SQLite. They include
  projections, DISTINCT, ORDER BY ... LIMIT, GROUP BY with HAVING, inner and left joins through an
  index, set operations, EXISTS, IN, correlated and uncorrelated scalar subqueries, and NULLs in every column.
- Each `WHERE p` divides into `p`, `NOT p` and `p IS NULL`, whose union must equal the unfiltered
  rows (ternary logic partitioning).
- Surveys with new seeds (80,000 queries) found no difference.

### Crash tests

Built for the log.

- **Damaged logs.** The log is cut at every byte, every bit of it is flipped, and a zero tail is
  appended. Recovery must give exactly the complete commits before the damage, or report corruption
  when complete commits follow it.
- **Sync failure.** The engine and the log reach files only through the `Disk` and `File`
  interfaces (`fs/disk.giz`). A test disk fails the first, the second, and each next sync of a
  workload in turn: log appends, checkpoint files, the directory. Each time, the engine must stop,
  refuse commits, and reopen with every synced commit, and with the commit in flight complete or
  absent (`tests/fs/fixture.giz`).
- **kill -9.** 1000 loops of SIGKILL during concurrent commits on a real disk lose no acknowledged
  transaction, and leave no transaction partly applied (`bench/db/reliability/crash.giz`, result in `bench/db/README.md`).

## 9. Text analysis

**Built** (analyzer version 1):

1. NFKC normalization.
2. Unicode full case folding.
3. Words split at Unicode letters and digits.
4. Hangul, Han, Hiragana and Katakana runs become overlapping grams of two characters. One
   character alone stays a token. Thus Korean and Japanese match without spaces (the CJK approach of Lucene).

The analyzer version is part of the schema of the column. A change of the version builds the index
again. The next analyzer is a Korean morphological analyzer (the model of Kiwi), measured against
this one on MIRACL-ko.

## 10. What GizDB does not do

- A second storage format, log, transaction mechanism or permission path for a new feature.
- Shared-disk cache coherence (RAC).
- Read locks. Readers never wait. Only writes and FOR UPDATE take row locks (§3).
- Its own query language next to SQL. SQL is the outside interface. The plan algebra is the inside interface.
- Silent staleness or approximation. A result states its watermark, and whether a search was
  approximate (`exact` is a query option).
- Hot loops moved into C. When Gizlang is slower, the language is made faster.

## 11. Native versioned journal

The Engine and the WAL carry a versioned journal (`db/core/journal.giz`): `changes()` and watches read it, and Kafka
producer outcomes (`db/core/journalstore.giz`) commit with their input.


### Positions

- `Position(shard, history, sequence, operation)` tells a committed transaction apart from its ordered operation.
- `COMMIT_END` is a completed transaction boundary.
- Positions from different histories cannot be compared. Promotion requires an explicit new snapshot or rebuild.
- Physical WAL offsets stay separate.

### Reversible recording

- Reversible recording is active when the engine has a history: a server's primary and `Access.local` name one
  before the engine opens.
- Its versioned appendix uses the WAL. The recovery and replication owner replays it.
- Older records replay normally. A historical journal request without reversible images fails with
  an explicit requirement for a new snapshot boundary.
- A key mutation of a RESP SET without options (`Engine.mutate`) records only its new row, as journal appendix
  version 4 does for a put without a before-image. Its earlier version stays in the table (MVCC), not in the WAL a
  second time.
- A shipped READ COMMITTED UPDATE that sets its columns from its key and constants is a key mutation too
  (`Engine.mutateShipped`): no transaction, sealed with the pass's other mutations as one commit. Its rows carry the
  before-images the statement read, so the record has journal appendix version 2, as the transaction's had. A batch
  holds rows with images or rows without, never both. A row lock, a unique secondary index or a changed read sends
  the statement to the transaction path. `pg_stat_gizdb.shipped_mutated` counts these UPDATEs.
- The legacy scalar `changes` result keeps its columns. It now refuses a limit that would divide one
  commit. Thus the next scalar-cursor request does not skip rows.

### Producers

`admitProducerOutcomeIn` keeps a window of 1 to 256 contiguous epoch, sequence and SHA-256 receipts, in the same
transaction as the input writes.

| Case | Result |
| --- | --- |
| Exact retry that is retained | Returns false. |
| Changed input | Conflict. |
| Expired retry | Fails. |
| Gap | Fails. |
| Older epoch | Fenced. |

- Version-1 metadata loads as a window of one receipt. An established window cannot become smaller silently.
- A new epoch starts at sequence one.
- Native durability cuts do not bypass the selected synchronous replication acknowledgment gate of the server.

Each receipt keeps an optional opaque acceptance result of a maximum of 128 bytes.

- An exact replay returns the result that was published first, also if the caller now computes a
  different candidate. It does not publish the input again.
- Version-3 metadata carries these outcomes. The Producer and ProducerWindow readers project their original fields.
- Version-1 and version-2 receipts have no result. Retries that depend on an outcome refuse
  explicitly. They do not invent an outcome.
- A receipt without a result keeps the retry semantics, and keeps the earlier outcomes.
- Results are admitted before the Catalog mutation. The only Codec reader limits declared binary
  lengths before a slice.
- This generic API does not attach the Kafka producer epoch and sequence on the wire.

### Kafka wire adapter

```sh
gizdb --dir DATA --port 5432 --kafka-port 9092 --kafka-topic events --kafka-tenant TENANT
```

This command enables an optional loopback plaintext listener on the authoritative Engine loop.

#### Configuration and authority

- The default is disabled (`kafkaPort: -1`). Port zero reports the held ephemeral port through `serveReportingPorts`.
- Kafka refuses to start with configured passwords, TLS settings or an SQL role catalog: it has no
  authenticated adapter.
- Native topic tables refuse SQL ACL and policy metadata. SQL guards the reserved `_giz_` namespace
  before grants, ownership and definer views. Superusers stay trusted.
- The enabled listener requires an explicit tenant that is not empty (a maximum of 256 bytes).
- Every native topic operation uses the Catalog scope of that tenant. Table names hash the framed
  identity of tenant and topic. Topics with the same name in different tenants have separate rows and offset sequences.
- The configured tenant of the listener is trusted startup authority. It is not wire authentication.

#### Protocol subset

ApiVersions advertises:

| API | Versions |
| --- | --- |
| Produce | 0 to 3 |
| Fetch | 0 to 4 |
| FindCoordinator | 0 to 1 |
| Metadata, ListOffsets, ApiVersions, InitProducerId, AddPartitionsToTxn, EndTxn | 0 |

- One explicitly configured topic has partition zero. Requests never create topics. Configuration
  makes the topic transactional (topic format v3, see Reserved offsets).
- Uncompressed magic 0 and 1 messages and magic 2 batches keep NULL and empty binary keys and values
  as different, and keep timestamps and record headers, in native Engine rows.
- Offsets and topic metadata commit together through Catalog MVCC.
- CRC32 and CRC32C, endian integers and varints use their SDK owners.
- An older Fetch format that cannot keep the stored fields fails explicitly.
- Compressed batches are refused. A producer's control batch is refused. A transactional batch is
  accepted only in a transactional Produce, and a transactional Produce accepts only transactional batches.
- Consumer group wire APIs, SASL, ListOffsets by timestamp and Kafka replication APIs are absent.
- This is a bounded protocol subset. It is not verified compatibility with Kafka 4 clients.

#### Idempotent produce

- Nontransactional InitProducerId v0 allocates a durable positive ID, scoped to the tenant, at
  epoch zero, through the Journal producer receipt and CAS owner.
- Produce 3 accepts one uncompressed idempotent magic2 batch for each partition, with acknowledgments.
- Kafka record sequences advance by the record count. Each canonical Journal batch receipt keeps the
  wire sequence and count, and the base offset that was accepted first, in its bounded outcome.
- Records, the topic frontier and the receipt publish in one Engine transaction.
- Exact retained retries return that offset without another append, also after a crash and reopen.
- Changed records, sequence gaps, expired receipts, unknown IDs and epochs that are not assigned
  are refused explicitly.
- Each partition keeps a maximum of 256 batch receipts. No hidden replay occurs outside that window.
- For an idempotent producer, epoch increase, sequence wrap and resumable InitProducerId versions
  are not implemented. Allocate a new ID before the signed 32-bit record sequence is exhausted.
- A lost initialization reply can leave an unused ID. A retry of InitProducerId safely allocates a new one.
- Modern Fetch builds record batches again without producer coordinates. Keys, values, timestamps
  and headers are kept. A v3 topic is read as Reserved offsets states.
- An exact Produce retry also uses the configured standby gate with a matching history. Its volatile
  confirmation resets on restart. A retained offset alone never bypasses that selected durability contract.
- The v0 InitProducerId layouts and wire error codes use the canonical
  [request schema](https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/InitProducerIdRequest.json),
  [response schema](https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/InitProducerIdResponse.json)
  and [Errors owner](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/protocol/Errors.java)
  of Apache Kafka.

#### Transactional producers

The broker serves the Kafka transactional producer flow on its one node:

- FindCoordinator with key type 1 (a transactional id) names this broker. Key type 0 (a consumer
  group) gets COORDINATOR_NOT_AVAILABLE (15).
- InitProducerId with a transactional id and its timeout calls `Txn.initializeReservedIn`: the first
  call binds a new PID at epoch 0, each next call increments the epoch, aborts the open transaction
  and fences the old epoch (47). An empty id is INVALID_REQUEST (42).
- AddPartitionsToTxn begins the transaction (`Txn.beginIn`) when it is not open. Only partition 0 of a
  transactional topic is accepted. A missing topic, another partition or a topic that is not v3 fails
  (3), and the other topics of that request get OPERATION_NOT_ATTEMPTED (55).
- Produce v3 with the transactional id stages each batch with `Txn.stageReservedIn`. The batch gets its
  offsets at once, and a read-committed Fetch does not show it until EndTxn.
- EndTxn calls `Txn.finishIn`: it publishes the batches with one commit or abort marker for each topic.
- Each request is one Engine transaction with the usual commit, sync and standby gate.
- The transaction deadline is wall-clock time (`System.time()`). An expired transaction is not
  aborted by itself: the next EndTxn abort or InitProducerId does it.
- The Kafka Java client sends Produce with acks=all, which needs a selected standby (`syncSlot`).
  Without one the broker refuses it (21), so a single node serves only clients that send acks 0 or 1.

#### Native transactions

`db/kafka/transaction.giz` adds, in the Engine, Catalog and Journal: a stable transactional identity
scoped to the tenant, a durable binding of PID and epoch, staged batches with reserved offsets (Reserved
offsets), and atomic publication or abort.

- `initializeReservedIn` keeps the PID and increments its epoch on reinitialization. It aborts the
  open transaction and fences stale callers. Epoch exhaustion refuses explicitly.
- `beginIn` uses a wall-clock deadline from `System.time()` that the caller gives.
- `stageReservedIn` keeps exact retry outcomes in the canonical Journal receipt window.
- `finishIn` publishes the staged rows, one marker for each topic and the terminal decision in one
  transaction of the caller.
- Old read snapshots do not change.
- The coordinator has fixed limits: 32 KiB of serialized state, 4096 staged records and 256 batches.
  It has no parallel WAL, database or retained open Engine transaction.
- The caller owns commit, local sync and the configured standby gate with a matching history. This
  includes unchanged retries of staging and publication.
- Expired transactions require an explicit abort. No automatic deadline worker is claimed.
- Only the terminal result of the current turn is kept. Retries of older turns refuse. They do not invent offsets.
- A PID bound to a transaction cannot bypass its coordinator through ordinary Produce.

#### Transactional framing

The only magic2 codec also exposes `transactionalBatch`, the server-only `controlBatch`, and bounded
typed `recordBatches` framing.

- Commit and abort markers are returned as metadata with no application records.
- CRC32C, integer framing and varints keep their SDK owners. Row admission checks the batch count
  before it allocates decoded records.
- Produce accepts transactional input only with its transactional id, and never control input.
- The supported marker layout uses the
  [ControlRecordType](https://github.com/apache/kafka/blob/4.0/clients/src/main/java/org/apache/kafka/common/record/ControlRecordType.java)
  and [EndTransactionMarker](https://github.com/apache/kafka/blob/4.0/clients/src/main/java/org/apache/kafka/common/record/EndTransactionMarker.java)
  owners of Kafka.

#### Reserved offsets

The native reserved-offset path uses the explicit `Store.upgradeTransactionsIn`. It adds one BYTEA
column for batch metadata and topic format v3, through the additive Engine schema owner.

- A clean transaction, physical-version credit, a mandatory scan and Catalog revision fences protect
  the upgrade. Payloads and MVCC versions are never dropped or built again.
- Engine DDL metadata is current, not versioned by snapshot. An old reader sees the additive column.
  Its v2 topic cut and its original record values stay intact through the canonical empty-fill projection.
- Old writer cuts conflict.

`Txn.initializeReservedIn`, `beginIn`, `stageReservedIn` and `finishIn` reserve actual offsets together with
native rows and the only Journal producer outcomes. Then they publish, in one Engine transaction,
bounded row decisions, one commit or abort marker for each topic, and the terminal result of the coordinator.

- The coordinator stores fixed descriptors of 32 bytes, not another copy of the payload.
- Reinitialization aborts outstanding reservations and fences their epoch atomically.
- Limits for each transaction: 4096 data rows, 256 batches, 32 KiB of serialized coordinator and
  1 MiB of encoded input. Each topic has a maximum of 256 pending producers.
- Commit and abort retries cannot append another marker.
- Timeout expiry still requires an explicit abort.

`Store.fetchVisibleIn` walks the shared offset index with row and byte credit. It keeps the identity
of batches and producers, and derives the last stable offset from bounded pending frontiers.

- A request inside a batch returns its original containing batch.
- Committed isolation stops at that frontier. It gives explicit aborted producer positions with raw
  transactional framing, which the consumer must filter. Markers stay typed metadata.
- Uncommitted isolation can return pending input.
- Ordinary Fetch refuses rows that carry batches. It does not discard identity or leak pending input.
- Fetch v4 reads a v3 topic through `fetchVisibleIn` with the request's isolation.
- Caller commit, local fsync and the acknowledgment of the selected standby with a matching history
  stay mandatory, also for retained retries.

#### Fetch, offsets and acknowledgments

- Requests and record payloads have a limit of 1 MiB. Each fetched partition has explicit byte
  credit and a maximum of 4096 records.
- New native topics have the canonical offset B+tree index.
- Older BLOB topics require the explicit trusted
  `Store.upgradeOffsetIndexIn(engine, tx, tenant, topic, maxVersions, backing)`, and a commit and sync of the
  caller, before Fetch. The refusal names this action. It does not silently go back to point probes.
  - The clean caller transaction limits all retained physical versions before the index is built.
  - Additive Engine schema publication keeps the physical columns and every MVCC version.
  - Mandatory table-scan fences refuse concurrent writes. Existing read pins stay valid.
- Fetch walks a maximum of 256 locations for each index batch, overlays the writes of the caller,
  and refuses missing offsets.
- Fetch 4 encodes one contiguous uncompressed magic2 batch. It preflights the exact frame, varint and
  header bytes before it builds the batch. Its meaning of timestamps, NULL, binary values and headers does not change.
- Fetch uses the parked client and wakeup owner for minimum bytes and maximum wait. Missing retained
  offsets fail, without a skip.

Produce acknowledgments:

- They use the local fsync and the receipt gate of the selected standby with a matching history.
- `acks=-1` without a selected `--sync-slot` returns INVALID_REQUIRED_ACKS without a mutation.
- When the selected standby times out, the connection closes with an uncertain outcome. The local
  transaction can already be durable. Examine it before a retry.
- The Produce request timeout limits the remote-receipt deadline, with the configured synchronous
  timeout of the server as a maximum. It never selects an asynchronous fallback.
- `acks=0` sends no response, and makes no claim of remote durability.
- No new log, consensus, automatic failover or producer receipt ledger is added.

#### Tests

- `tests/db/kafka/run.giz` contains the contracts for a real disposable child listener and wire, SQL
  visibility, crash and reopen, record preservation, CRC, acknowledgment refusal, ownership and Fetch wait.
- The fixture also stages a real acknowledgment of a designated standby, immediate visibility on the
  receiver, SIGKILL of the primary, crash, reopen and promotion of the receiver, and an uncertain
  timed-out write with exact checks of local durable reopen.

## 12. Native storage resources

Logical credits in this section are admission limits. They are not measurements of
allocator use or RSS.

### Budgeted readers


- It reserves before I/O. It admits the conservative peak of the raw window and the joined payload.
  It releases live credit on the mutable `close`.
- Independent views reserve from the same ledger.
- A credit refusal is an explicit I/O status. There is no fallback to a smaller window.
- This primitive is not connected to every Engine and Log caller yet. It does not admit decoded
  rows, resident tables or transaction operations.

### Transaction encoding

The private follow-on for transaction encoding uses canonical `EncodedParts` of 64 KiB, Readers of
operation-offset ranges and the frame CRC owner.

- Commit emits header, operation and Journal pieces, without a join of a full transaction body.
- Journal changes use the canonical bounded encoder before allocation.
- Savepoints keep their encoded cut across chunk boundaries.
- This removes contiguous copies of a commit. It does not remove the residency of the full
  transaction: offset and own-write metadata, pending pieces and decoded single operations still
  need shared admission and spill.

### Admitted WAL opening
`Engine.openOn(disk, repair, Option.some(context))` explicitly admits WAL header and CRC
`Engine.openWithResources(disk,repair,scope,windowBytes)` explicitly admits WAL header and CRC
windows, recovery frame headers and replay Readers, through the same actor scope of the caller.

- It carries no Ref in the returned copyable Engine snapshot, and makes no limit or ledger.
- An admission failure keeps its I/O status, closes owned files and releases leases. It never
  selects the legacy path without admission.
- Inspection stays read-only. A writable reopen still fsyncs before it publishes its recovered cut.
- Legacy open APIs keep their earlier behavior. This WAL-only primitive does not admit the residency
  of manifests and segments, decoded values or transaction spill.

### Scoped temporary pages

Temporary pages also expose explicit APIs scoped to an actor: `tempPagesScoped`, `appendScoped`,
`nextScoped`, `readScoped`, `sealScoped`, `closeScoped`, `File.readPageScoped` and
`LeasedPage.closeScoped`.

- They use the `Ref<Resources>` of the caller. Admission is stored synchronously before I/O.
  Finalization loads the same issuer again.
- The serial `mut Resources` APIs stay available. They share the physical read, CRC, short-write
  and rollback, and delete implementations.
- Each arena still has one writing owner. A scope can admit independent concurrent work, but copied
  arena values are not independent writers.
- A failed rollback keeps the temporary credit and poisons the arena. Cleanup releases the credit after the delete.
- These primitives do not spill Engine transaction operations yet.

### Journal image sections

Commit keeps the Journal header, the offset directory and the image chunks as different frame
sections. It uses the same CRC and WAL queue owners, and does not pack all image bytes into a second chunk set.

- The streaming recovery fixture now enables the recording through the real subscription owner. It
  verifies all 2,048 binary after-images, operation positions, partial-page cuts and the final commit
  boundary, before and after reopen.
- Earlier versions of that fixture wrote history but did not enable the recording. Thus they
  covered only operation replay.

### Pending WAL storage

`pendingStorage(sys,scope,pageBytes)` makes an admitted temporary arena for pending WAL. The actor owns it.

- `Engine.commitWithStorage(tx,storage)` prepares the same canonical frame as `commit`. It admits the
  full scratch frame, then publishes its row and catalog changes and its clock.
- Engine snapshots keep only the identity, the generation and the byte-range coordinates. The
  resource scope, the file and the arena stay external.
- `syncWithStorage(storage)` verifies and drains one admitted page at a time to the same WAL, fsyncs
  it, and advances `committed` through the same group-commit owner.
- Then the scratch is truncated and its lease is rotated. This fences old aliases.
- A scratch reset failure explicitly reports the cut that is already durable locally, and keeps the
  temporary credit until the delete.
- Scratch never certifies standby receipt or ACK.

Rules:

- Legacy write, sync, checkpoint and close paths refuse a live stored range, and name the required
  API that takes a context. A wrong context refuses before I/O.
- Earlier inline pending records drain first.
- A failed staging leaves the new rows and clock not published. An uncertain rollback poisons the
  context and stops the Engine.
- A WAL, read, CRC or fsync failure does not advance the acknowledged durable cut.
- The log append owner uses the canonical exact-write completion. Thus a short write cannot advance the LSN silently.
- This is opt-in spill of pending WAL only. The SQL and RESP actor adapters do not propagate the
  context yet. Staged Ops, own-write directories, decoded values and resident Table state stay
  separate gaps of admission and locality.

### Larger than memory

One engine holds every table: segments, the delta, one primary tree and the secondary trees, with the WAL
and segment files on disk. Memory bounds it the way a buffer manager bounds
pages (Umbra, LeanStore), so a resident read stays a pointer read and data past memory stays usable.

- **Budget.** An actor's tables keep at most half the memory free when it starts
  (`StorageContext.dataBudget`, `Engine.dataBudget`). Within it nothing changes: no lookup, pin or
  check besides one tag test on a segment and one on an index page.
- **Columns.** A segment opens cold: its columns stay in its immutable `seg-` file. A query reads back only the
  columns that it names, one block each (`ColdHome.column`), into cells that every view of the table shares. The
  cells keep columns while the budget holds them. Past it, a column serves its query and is not kept, and the next
  `Engine.evictCold` lets go of the columns least recently read to make room for it: each column of each segment by its
  own last read (`ColdHome.stamps`), so the columns of the statements that run now stay. Past the budget,
  `Engine.evictCold` also moves the oldest resident segments to their files. Zones, `begin` stamps and text and
  vector indexes stay resident.
  The budget shrinks by what the process lacks: where its room (`coldRoom`: the room left under `GIZLANG_MEMORY_LIMIT`, or in
  the cgroups and on the machine with other processes counted, whichever is smaller) is under 1 / `ROOM_RESERVE_SHARE` of the
  budget, the cells keep no new column (`ColdMeter.refuses`), the engine ticker calls `evictCold` at once, and it aims lower by
  the difference. `pg_stat_gizdb.cold_held` shows the bytes the cells hold. Wide rows reach the budget in few rows, so a check also follows each 1 / `EVICT_CHECK_SHARE` of the budget in
  applied operation bytes, a delta holding that much flushes to a segment (`Engine.flushWide`), and a load run is cut by bytes
  (`Engine.flushRows`), not only at `FLUSH_ROWS` rows.
- **Keys.** One B+tree a table (`Table.primary`) maps each key to its newest version, wherever that version is:
  a cold segment's rows stay in it. A key run (a load's segment, its keys in order) is a leaf of its own key range
  with no entry a row; a lookup the tree misses halves the runs (`Table.locate`). A version names the one it replaced
  (`Table.prevAt`), so an older snapshot walks back from the newest version (`Table.seenAt`), as an undo chain.
- **Index pages.** Still past the budget, the primary and each secondary tree move leaf pages to their own
  append-only `spill-` file (`Ordered.evictLeaves`). The directory keeps a stub with the page's
  offset, so each copy of a tree reads back the version it names; a write makes the page resident
  again. Spill files are scratch: open removes them and rebuilds the trees from the segments and WAL.
- **Open.** A database larger than memory opens: it reads the segment heads, not the columns.
- **Large writes.** COPY, a large INSERT, and an autocommit UPDATE or DELETE of 100,000 rows or more
  (`Conn.bulkRewrite`) run as one chunked transaction: each chunk is one WAL group record applied to the group's copy of
  the tables, whose full segments go to their files while the load is past the budget. A crash or failure before the
  commit leaves no row visible; recovery and a standby apply a group only once it is sealed (`Engine.replayGroup`). A
  chunk's key check leaves out the load's own chunks, and when every key of the table is in a key run it searches only
  the runs whose range meets the chunk's (`Table.holdsAny`). The load's keys are checked against each other once, at the
  commit (`Engine.chunkedDistinct`): none when its runs are apart (`Table.runsApartFrom`), else by their hashes, split
  by their top bits and sorted on every core. The COMMIT writes each staged file as a chunk of one WAL group at the
  commit's sequence and attaches the runs (`Engine.commitStagedWithContext`); it fails with SQLSTATE 40001 when the
  table or the catalog changed since the COPY. ROLLBACK or the connection's end lets the files go. Any other statement
  of the transaction first makes the staged rows the transaction's own writes (`Conn.materializeStaged`). A load's key
  runs are kept in low-key order (`Table.runOrder`): a key lookup halves them, then compares a run's rows part by part
  (`Table.runSearch`). An INSERT ... ON CONFLICT on the primary key loads the same way (`Conn.stageConflicted`): a
  chunk's new keys are one run, and the rows it changes are another, which ends the versions they replace as it attaches
  (`Table.supersede`); its WAL chunk carries their before-images (journal appendix version 6), and replay finds those
  versions again by their keys.
- **Queries.** A resident scan reads the cold columns that it names after zone maps prune the segments: up to 16 tasks
  of one thread read the blocks at once (`COLD_READERS`) and every core decodes them. A scan that would read back more than half the budget fails with
  `SCAN_OVERFLOW`, and the statement runs again on the stored path, which streams blocks. A sort that keeps at most 1,024
  rows (LIMIT and OFFSET) over a scan reads the columns of its keys and filters first, keeps the top rows by their
  locations, then reads only those rows whole, a part at a time (`lateTop`, as DuckDB materializes a top-n late). A cold
  part decodes only those rows of each column's block (`decodeRows`: a bit-packed frame, a run, a double, an ALP digit, a
  dictionary code and its string are found by position). A sort with a LIMIT whose first key is an integer, date or time column
  of its scan bounds that key first (`zoneCut`, DuckDB's Top-N dynamic filter): the zone maps' best blocks, about a
  thousand rows a kept row, set a bound, and the scan reads only the blocks that can pass it. When those rows hold the
  whole top the sort is done; else it reads four times as many. A scan holds only the parts whose zones may match, so
  its reads, its kernel's lanes and params and its tasks cost the parts it reads. Sorts, joins, aggregates and windows
  past their statement credit spill to temporary pages.
- **Files.** A merge's input segments and the files of an aborted load are removed one checkpoint
  later (`Engine.retired`): a thread's view or a cold read may still read them. Open retires the
  segment files the MANIFEST does not name.

Limits: every key of a resident segment, every `begin` stamp of an unsettled segment and each tree's
inner pages stay in memory; a scan reads a named column of a segment whole, not only the blocks that its zone maps
keep.
A `MANIFEST` of the removed persisted-root format (`GIZROOT1`..`GIZROOT3`) is refused: recreate that
database.

### Shared serial and actor Store access

`ResourceContext` in the Disk owner selects serial Resources or a scoped reference to an issuing
actor. It adds no ledger or policy.

- PagedOrdered lookup, rank, cursor and path-copy mutation, and selective column reads of
  SegmentRef, run one common algorithm with this capability.
- The original serial APIs delegate, and return every changed reservation, also during error cleanup.
- Actor capabilities never keep loaded Resources across I/O. PageCache publishes the admission first,
  and loads the same issuer again for finalization.
- The mutable cache itself stays under one caller owner. It is not copied independently between concurrent lanes.
  To serve requests while a miss reads, the owner uses `admitContext`, runs `CacheRead.read` in
  another task and gives its result to `completeContext`. Each read in flight holds its own slot,
  so hits and other misses go on. Another miss of the same page, `invalidateTail` over it and
  close refuse until the read completes.

Cursors:

- Cursor traversal advances its parent position only after the page admission and read succeed.
- A failed child descent restores the bounded path. Thus, when the memory limit is restored, the
  cursor continues at the same unconsumed key.
- Close attempts by a foreign issuer keep the live cursor and column contents.

Evidence:

- This includes point, rank and scan, old and new index roots, selective column reads and complete cleanup.
- Logical reservations do not establish allocator or RSS limits, or automatic default SQL spill.

### Actor-owned persisted read context

#### Resource policy

- `resourcePolicy(memory,temporary)` derives the internal geometry of leases, cache, pages, manifest
  and queries in the common Store resource owner.
- It accepts the canonical allocator ceiling and explicit temporary and disk credit. Zero temporary credit stays zero.
- `resourcesForPolicy` refuses edited geometry, and issues through the Resources factory.
- These are ceilings. They are not measured free memory or RSS.
- The next routing step is issuance by the PG actor.

#### Open with context

- `openWithContext` admits a bounded format prefix. Then it selects the resident or the persisted-root recovery owner.
- Each owner validates the checksum and framing of its full manifest. A failure never falls back to the other format.
- One StorageContext opens one Engine. A second open refuses before I/O.
- Legacy resident manifests and segments still use their eager decoder. Replay of a stored WAL tail
  without context stays explicitly not supported.
- Source fixtures cover empty, resident checkpoint, persisted, repeated open and malformed manifests.

#### Writes with context

`putWithContext`, `deleteWithContext` and `commitWithContext` stage a transaction's rows in the
resident tables and its WAL record in the pending storage.

- A commit success is a visibility sequence, not a durability acknowledgment. `syncWithStorage` must cover it.
- `openRecoveryWithContext` replays the WAL commit by commit, a sealed chunked group whole
  (`Engine.replayGroup`). A read-only inspection writes nothing.
- `checkpointWithContext` syncs and checkpoints: new segment files, then the MANIFEST, then the logs
  recovery no longer reads.
- PITR uses the canonical checked record header. A checksummed frame with a truncated flagged header,
  an invalid sequence or a negative appendix coordinate cannot silently get wall time zero.

#### Transaction column scans

`scanTxColumnsWithContext` scans the original registered cut plus the latest own PUTs. It suppresses
superseded cold rows and staged DELETEs.

- It projects only the requested physical columns plus primary-key metadata. The canonical Value
  decoder skips excluded own binary payloads.
- Only `StoredTxPoll.done` certifies the end. An empty poll that is not terminal is bounded work, and yields.
- Rows and cursors keep their issuer grants until they are explicitly closed.
- A staged mutation or savepoint rollback invalidates a continuation through the canonical Ops
  revision. Exhaustion refuses before it changes encoded operations, and fences the commit.
- Scope reads record the table-read conflict check.
- Written regressions cover own replacement and deletion, old snapshots, savepoint and serializable conflicts, refused
  preparation, cold replay, checkpoint reopen, and SIGKILL of a real child process on both sides of group sync.
- Inspection of the prefix and length of an encoded cell is not exposed yet. Callers must not
  hydrate an excluded payload only to examine its kind. This is a remaining requirement for the
  inspection of cold containers.

#### Stored queries

`Conn.retainStoredRead` binds through the SQL authorization and row-policy owner. Then it gives
checked actor acquisitions to the canonical `FiniteInput` pipeline.

- The cursor requires its original live read-only transaction and the bound catalog.
- The cursor state and each returned delivery have independent physical grants. Callers close deliveries explicitly.
- `Conn.statementStored` shares the transaction, abort and rollback owner of the ordinary statement,
  and returns a leased `StoredStatement`. Its collected result has explicit row and byte limits.
  Larger results use the finite cursor.
- The initial strategies support one SCAN with a FILTER on rows only, and a PROJECT of columns and
  constants. Other operators, mutations and session effects return checked errors.

#### Storage context

`storageContext(system, disk, scope, cacheSlots, pendingPageBytes)` uses the `Ref<Resources>` issuer
of the caller, and owns one mutable PageCache plus the PendingStorage.

- The Engine stays the logical MVCC snapshot. No issuing reference or cache enters the Engine or
  read-only worker snapshots.
- `getWithContext`, `scanWithContext` and `nextWithContext` use the same Table traversal and
  visibility body as their serial versions, through the shared ResourceContext page APIs.
- Do not copy the mutable cache or context of the actor across concurrent lanes and publish it again independently.

Close rules:

- Every returned row and scan must be closed with the owning ResourceContext. A close by a foreign
  issuer refuses before it drops the live values or cursor.
- Scoped MANIFEST and Log reads load the issuing ledger again at the admission and finalization
  boundaries. This keeps unrelated claims made during I/O.
- MVCC delta rows are admitted before binary and vector payloads are materialized. They take part in the same Table scan.
- The read-only API does not do stored writes or WAL replay. Those require the explicit writer and recovery methods above.

#### Stored blocks

The source implementation `Table.nextStoredBlockContext` accepts selected schema columns, row credit,
work credit for invisible rows, and byte credit for prepared output.

- It visits a maximum of one physical column block or delta page for each acquisition.
- It decodes each unique requested immutable column one time, plus a timestamp carrier when necessary.
- Column gather prepares typed vectors directly. Duplicate projections share the source decode.
  Added columns keep their schema fill.
- Delta text output builds only the selected dictionary, and keeps no unrelated entries.
- Visibility still uses the canonical begin and end interval and the persisted ends index for each candidate row.
- `StoredBlock.ids` keeps monotone physical source positions, also for invisible candidates. Its
  `scanned` counter measures row work.
- Zero visible rows return a bounded block, to yield. Only the physical end returns None.
- Output and decoded-column leases stay with the block until `closeContext`. Failures leave the scan position unchanged.
- This is conservative admitted credit, not measured RSS.

### Prepared SORT input

`beginDiskSortInput` opens the external SORT strategy for a source that an actor prepared.

- `pushDiskSortInput` accepts a dense bounded Batch. It checks the actual column metadata, widths,
  counts and IDs, and the input overlap, before key evaluation or a file mutation.
- Planned type hints do not cast raw or computed cells.
- The same run encoder keeps heterogeneous typed values and canonical equality and order keys, also
  raw binary, without a new comparator or codec.
- The source order gives the stable tie ordinal. The acquisition actor must keep its authorized plan
  and context and its MVCC pin, issue batches in that order, and keep and drop their original grants around each push.
- The builder holds no Engine or issuing Ref, and does no legacy table scan.
- `finishDiskSortInput` uses the merge and seal owners and the LIMIT/OFFSET owners. Then it transfers
  one ordinary DiskFinite cursor.
- Cursors that are not finished cannot deliver rows. A failed input or merge cannot finish.
  `closeDiskSortInput` keeps the cleanup owners.
- Prepared ingest and legacy finite acquisition now share one encoder and merge body.

#### Mixed aggregate input

`DiskMixedAggregateInput` sends the dense batches of the original child into every canonical
mixed-aggregate descriptor SORT.

- The actor acquires and projects that child one time. Begin, push and finish never acquire it again.
- Its Context APIs use the same issued budget and resource owner for all builders.
- Begin publishes an explicit cleanup slot before builder admission. A rollback failure leaves that
  poisoned slot with the caller.
- A partial push or finish poisons the acquisition, and keeps both unfinished builders and sealed sources for close.
- Finish transfers sources only after all descriptors seal, in their original descriptor order.
- The bounded descriptor shell has an explicit preflight credit, included in the simultaneous phase
  and overhead accounting of the execution planner. It stays held until its payload is dropped.
- Foreign and stale contexts refuse before mutation.
- This boundary does not authorize the substitution of a bound context, relax the verification of
  prepared sources, or certify the MVCC pin of an actor.
- The MODE scenario adds trace receipts of the child that occur one time only, exact rows for each source, and contracts
  for rollback, partial failure, issuers and cleanup.

#### Runs

- Rows collect as columns until half the statement's state holds them: their columns (`chargedBatch`) and 64 bytes
  for each key and row. A batch past an eighth of the state goes in halves.
- A run's keys are made on the pool, 4,096 rows a task. The keys sort on every core (`orderParallel`), with the input
  position as the last key, so the sort is stable. Each column then gathers on a core of its own.
- A run page holds each column's type and its values in the segment block codec (`columnBytes` in `db/core/segment.giz`;
  a mixed column as values, a column nobody reads as nothing), then each row's keys and input position. A page holds at
  most `pageRows` rows and about `pageBytes`. A page past the arena's largest goes in halves.
- The merge keeps a heap of its runs by their heads, and one decoded page of each run. Rows that come from one page in
  a row go out as one slice of its columns. It merges at most 64 runs a pass, on one core.
- The reader gives slices of a page's columns. OFFSET passes whole stretches of a page. No row becomes values.
- Prepared input admits row and key slots before key evaluation. Single-row layouts that are not possible refuse
  before a temporary file is acquired.

### Prepared GROUP and JOIN input

`diskGroupInputPlan` and `diskJoinInputPlan` are the only owners of the prepared sort layout of the
operators. Actors can fill those prototypes through `DiskSortInput`. Then they give one or two sealed
cursors to `diskGroupedFromSort` or `diskJoinedFromSort`.

- These receivers use the same GROUP allocator (with sparse scalar and statistical states), JOIN
  admission, and bounded row, probe and finish kernels as finite acquisition.
- They do not capture Env, open another Engine scan, keep an input directory of the population size,
  or build sorted rows again.
- A successful admission takes the given handle vector. Any refusal leaves the cleanup and the source
  ownership with its caller.

Admission requires:

- an untouched sealed equality cursor;
- no LIMIT, OFFSET or pending delivery;
- the same budget and page configuration, and the same issuing Resources owner;
- an exact bounded `physicalPlanBytes` identity with the canonical sort prototype.

Rules:

- This is one conservative admission pass. It is not optimizer equivalence, and not an unbounded plan export.
- Each definition has a limit of min(stateBytes/32, 16 MiB), 16,384 nodes and depth 64. Large inline
  VALUES definitions can refuse explicitly. Stored-source prototypes do not include their input population.
- Group evaluation inherits the Ctx of the prepared source.
- Prepared JOIN also requires VM `sameRowContext`: all independent Dialect flags and the scalar
  bindings of the statement must agree. Plain row contexts with parameters, outer values, subqueries
  or library bindings stay not supported by that structural check.
- A dialect name alone cannot establish a compatible meaning of expressions.

#### Statement contexts

- VM `issueStatementContext` captures the completed Ctx in one managed authority.
- `PreparedRowContext` carries a plain context or that issued statement. `preparedRowContext`
  gets the only expression context.
- Issued contexts agree only when they have the exact same opaque identity. Equal parameter bytes or
  callable metadata do not establish identity. Function values are never compared.
- `beginDiskStatementSortInputContext` uses the SORT allocator with this authority.
- Mixed acquisition accepts the same explicit PreparedRowContext. Its prepared receiver also receives
  the expected context of the actor, and refuses the replacement of one or all source tokens.
- JOIN requires a common source authority.
- Merge, slice, key evaluation, cancellation and recursive restaging keep that same meaning.
- The native issuer owns the lifetime of the authority, and must not overwrite its captured bindings.
  It stays different from actor authorization and from the pinned MVCC cut.
- Fixtures cover copied bindings, the evaluation of callables, parameters, outer values and subqueries, identities that
  agree, refusal of substitution, and cleanup.

#### Scope

- The acquisition actor still owns authorization and the common pinned MVCC cut across sources. A
  pure DiskFinite carries no Engine transaction token. This API does not certify a snapshot only
  from equal Ctx values.
- Default Session and PG dispatch stay the responsibility of the actor integration owner.
- The final SORT merge stays synchronous. Bounded output fanout and row cadence come from the admitted operators.
- They also pin refusals of context, prototype, issuer, phase and resource without a loss of ownership.
- Logical grants are separate from RSS.

### Close the storage context

1. Close the Engine through `closeWithStorage(context.pending)`.
2. Release its logical snapshots.
3. Close the StorageContext.

- Cleanup refuses open transactions and pending WAL that is not confirmed. It keeps the first cleanup error of PendingStorage.
- This API is not a second reader engine, and not a completed server with bounded memory.

### Selected point reads and BLOB prefixes

Selected point acquisition uses `Table.getStoredColumnsContext` and `rowStoredColumnsContext`, for
resident and stored backing. Full point calls delegate to the same owner.

- The projection attests primary keys and exact MVCC intervals with only the required columns.
  Unrelated BLOB, vector and dictionary payloads are omitted.
- The actor keeps the registered snapshot and the overlay of staged writes certified by revision. A
  pure prepared cursor is not an independent pin.
- The same selected-block entry point now accepts resident segments and Builders, without `parts()`
  or full-row hydration, with bounded candidate work and a typed output with its own lease.

- Stored BLOB inspection follows the Segment directory and the CRC page cache. It checks encoded
  offsets, NULL and MVCC framing, and copies only the requested prefix. It bypasses the decoded Column
  arena and the allocation of the offset vector.
- Physical encoded pages are still read and checksummed, and limited by the stored page limit. This
  does not claim I/O smaller than a page, or checksum work in O(1).
- Prefix and result leases stay owned by the caller until close.
- Schema fill, an empty BLOB and NULL stay different. Failures publish no projected row or prefix.

### Transaction key cursors

The Context transaction API uses one ordered version-key envelope for resident and persisted backing.

- `scanTxKeysWithContext` accepts selected columns and an optional last logical key.
- `nextTxKeyWithContext` returns `next`, bounded candidate work, and a separate `complete` receipt.
- Deleted or invisible keys can advance `next` without a row.
- A new transaction seeks after that key under its new registered cut.
- A mutation or savepoint rollback invalidates a transaction cursor.
- The latest own PUTs and DELETEs are merged through the transaction projection of the Engine.
- All returned grants require the original issuer.

### Drop and recreate

`Engine.dropWithContext` stages `Op.drop`. SQL DROP uses the statement actor admission and the same
PreparedCommit, WAL and replay publisher.

- The prepared directory removes dropped incarnations after the row stages finish.
- A DROP and CREATE in the same transaction starts a new physical Table. It never adopts the old payload or index roots.
- No physical files are deleted. Old pure Engine and Table snapshots and their registered cuts keep their original roots.
- Current catalog lookups keep their meaning. Live cursors bound to the catalog invalidate on a catalog change.
- This does not add a separate MVCC catalog for schemas, or a tombstone store.

Catalog revision reads:

- Buffered row, schema and drop mutations record one mandatory read of the catalog revision. Thus
  mutations across DDL refuse, also at weak isolation, and also for an absent key in a recreated
  table. Ordinary row concurrency without catalog changes does not change.
- Dropped physical files, and failed files that were not published, stay until a later canonical
  retention and reclamation implementation proves every relevant cut safe.

### Replication receive with context

Replication with context receives bounded raw WAL chunks with
`Engine.receiveWithContext(sys,lsn,data,storage,policy)`.

- The transport continues at `Log.received`. Only the returned `Log.end` is a durability
  acknowledgment. It comes after complete-frame CRC validation, canonical replay and a successful fsync.
- The WAL itself holds partial bytes. An incomplete tail refuses local write, sync, rotation,
  checkpoint, backup and archival. Reopen uses the torn-tail repair and the complete-frame replay owner.
- Wrong transport positions refuse without a mutation. A write, CRC, replay or sync failure stops the receiver.
- A complete frame that was not acknowledged can recover on restart. Do not describe it as rolled back.
- Announced frame bodies and decoded work obey the query budget of the actor.
- CREATE, compatible reshape of retained columns, and DROP and recreation replay use the same
  canonical physical writer and catalog validator. Type and layout changes that are not supported
  still refuse before publication.

### Empty physical databases and CREATE

A writable `openRecoveryWithContext` publishes an empty physical ROOT3 cut through the checkpoint
publisher, before it returns a database with no tables.

- Its metadata lease stays the authoritative ownership of the physical format by the actor, also before CREATE.
- No additional Engine mode, initialization transaction or WAL is added.
- Catalog objects and the certified commit cut survive an empty legacy migration.

CREATE:

- `defineWithContext` stages a new schema through the original transaction operation owner.
- Commit prepares durable empty primary, version-end and declared btree secondary roots. Then it uses
  the row and unique-constraint writer before the original WAL and group-sync publication.
- The same adapter replays CREATE and its initial rows after restart or receive.
- Compatible schemas and DROP and recreation use the adapters above.
- `contextTable` exposes registered and staged metadata. `lockRowsWithContext` shares the resident
  lock and historical-change checks with admitted physical reads.

Evidence:

- Backup with context keeps this empty physical manifest. It does not write a resident one.

### Prepared commits

`commitWithContext` prepares a commit, then publishes it. The prepared commit keeps the physical
staging and the actor admission, and it is used one time.

- Retirement closes the stages and clears the payload before it releases the admission.
- A cleanup failure stops the Engine. It never restores write authority.
- Publication checks the clock, catalog, log and root metadata before the pending WAL and the
  logical publication run. A changed cut refuses before either changes.

### Allocation and retained reads

- `allocateWithContext`, `lastChangeWithContext` and `refreshWithContext` use the Table metadata,
  version-chain and validation owners.
- `retainReadWithContext` registers the original cut and the staged-read image of a suspended query
  in the original open-snapshot map.
- The retained read scope cannot refresh or commit the writes of its parent. It closes through ordinary `abandon`.
- Commits may interleave with an open retained stored read. The caller continues with the
  StorageContext that `retainStoredRead` leaves it (`readAll`, db/access.giz);
  a copy taken before the retain has a stale page cache, and a commit through it fails with ESTALE (-116).
- Written roots fixtures cover cut retention, preservation of the staged image, allocation exhaustion, preparation
  refusal and cancellation, and durable publication.

### Transaction-visible Context index reads

- `lookupTxWithContext(tx,name,index,values,storage,columns,maxKeyBytes,maxStateBytes)` makes an
  equality-prefix cursor, and records the canonical index-entry dependency, also for empty results.
- `scanTxIndexWithContext(tx,name,index,storage,columns,spans,descending,maxKeyBytes,maxStateBytes)`
  uses the half-open `Span` encoding. For ranges, it records a table-scan dependency, to be safe.
- `nextTxIndexWithContext` returns selected raw candidates, bounded physical work and a separate end.
  The SQL owner applies predicates, RLS and survivor limits.
- The same APIs serve resident and persisted btree indexes.

The cursor:

- It keeps the original pure Table and MVCC cut, and the transaction operation revision.
- A bounded, admitted projection of the latest own PUTs is sorted one time. Own deletes and
  replacements suppress committed candidates.
- Physical candidates use the canonical secondary tree and the selected-column decoder. Reverse
  traversal shares the original paged cursor algorithm.
- This first adapter seeks the secondary tree again for each candidate (O(log N) each). It does not
  build or scan the relation again.
- Explicit refusals of state, key and row credit keep the unconsumed logical bookmark.
- Mutation, savepoint rollback, closed pins, schema and catalog changes, and foreign issuers refuse continuation.
- Staged row IDs stay `-1-opOrdinal` at the original transaction revision. Physical IDs keep their
  canonical locations. Full and projected point reads and binary-prefix reads use that same staged
  operation location. They do not collapse distinct own rows to one shared `-1` identifier.

Evidence:

- Runtime assertions cover both backings, the latest own writes and tombstones, reverse ranges,
  complete prefixes, conflicts on empty results, old-cut visibility across checkpoint, resource
  refusal and resume, rollback invalidation and reopen. They have not run yet.

### Native bounded WAL groups

`operationBatch` and `Ops.pushBounded` admit each PUT and DELETE batch before the original operation
codec allocates it. `beginChunkedWithContext`, `stageChunkedWithContext` and
`commitChunkedWithContext` keep one registered current MVCC and catalog cut while they prepare
bounded batches through the StoredStage owner.

Requirements of this initial native API:

- one drained versioned table;
- reversible Journal enabled;
- a drained, healthy local primary;
- the canonical resource policy of the actor.

DDL, groups across tables, and local publication bound to consensus refuse explicitly. This API does not give
arbitrarily large transactions or capacity at TB scale.

#### WAL layout

- The original WAL contains ordered bounded chunk frames and one final boundary with SHA-256 and count.
- The final boundary is fsynced before the only publication of the Engine root and clock, and before the local durable receipt.
- Intermediate physical frames are not committed cuts, and not replication ACKs.
- Ordinary writer aliases share the managed writer revision of the Log. The group reservation uses
  it before rollback-handle I/O. Stale copies from before the reservation cannot overwrite or close the current handle.
- A healthy abort that is not sealed restores the original tail.
- A final write or sync failure stops the Engine, and keeps the uncertain evidence for canonical
  reopen. It is never reported as a successful rollback or as remotely confirmed durability.

#### Readers

- The same semantic commit dispatcher serves cold reopen, partial receive of Context replication,
  `changes()` and WAL-only verification.
- The WAL-only verifier does not give full SQL constraint verification of cold tables, or grouped salvage.

#### Legacy feed

- The legacy feed in memory keeps a maximum of the `queryBytes` admission of the native policy and the `FEED_KEEP` operation cap.
- A group that is too large expires that legacy prefix only at final publication (`feedStart`
  advances to the group sequence). Reversible durable Journal images and rows stay intact.
- Consumers receive the explicit trimmed outcome. They do not receive a partial group or a silent Journal skip.

#### Evidence

- Written contracts cover before and after images of repeated keys, old-cut visibility, abort and
  reopen, groups larger than the credits of one query, explicit feed expiry, uncertain outcomes after
  a false fsync, partial receive, and SIGKILL of an owned process at staged, sealed and published cuts.

### Query-owned physical reads

Query-owned physical reads use the original Engine `getContext`, selected point and BLOB, block and
transaction cursor methods, with a `PageCache` and a `ResourceContext` that the caller owns.

- A query issues its cache under its execution fence. An actor cache cannot get a new label.
- The actor `*WithContext` methods delegate to these same bodies.
- A cache that does not agree, or is closed, refuses before cursor movement, decoding of own rows or physical I/O.
- Cursors and returned rows close through their original context.
- Checkpoint publication stays the canonical `checkpointWithContext` owner of the actor. It keeps
  registered query cuts and immutable physical roots.

### Checkpoint plans

ROOT3 checkpoint scheduling uses the original actor publisher `planCheckpointWithContext` and
`finishCheckpointWithContext`.

- Planning settles the WAL, and admits an immutable manifest for the original root cut, the Engine
  WAL identity, the actor, the history and the policy. It does not publish, rotate or reset changed rows.
- `StoredCheckpointPlan.close` cancels or retires that plan. It clears its manifest before it releases the original grant.
- A cancelled, foreign or stale plan refuses, without a change of an admission or conflict into a storage failure.
- `checkpointWithContext` delegates plan, finish and close, and keeps the original failure.
- Publication still does the original rotation, MANIFEST replacement and directory sync. It keeps
  live query cuts and never reclaims pinned roots.
- A false sync stops publication without an acknowledgment of the checkpoint. It does not promise a
  rollback of a MANIFEST that was already written.

### Projected transaction blocks

Native projected transaction scans use
`nextTxBlockContext(tx,scan,cache,context,maxRows,maxWorkRows,maxBytes)` with the original registered
cut, the selected columns and the own-write revision.

- Each poll acquires one physical Table block, masks keys that the transaction replaced or deleted,
  and gathers through the Column kernels.
- Physical row IDs do not change. Staged IDs are `-1-operationOrdinal`, tied to the original transaction revision.
- `nextTxBlockWithContext` is the actor delegate. Row polls delegate to the same algorithm.
- The returned `StoredTxBlockPoll` has `block`, `scanned` and `done`. A block with zero rows, or an
  absent block with `done=false`, is bounded work. Only `done=true` certifies completion.
- Every returned block must close under its original Context before its grants are retired.
- Row, work and prepared-byte credit cover selection, gather, key scratch and staged decoding.
  Decoded physical source columns keep their separate original I/O grants until the block closes.
- Variable equality keys use the only `keyValueBounded` encoder, admitted before the source key is materialized.
- A failed admission keeps the original cursor position. Foreign or closed query capabilities refuse
  before I/O or cursor movement.
- Written resident and cold fixtures cover vectorized physical and staged output, exact IDs and rows,
  bounded empty work, refusal and retry, invalidation by rollback revision, key credits and cleanup.

### Physical planning admission

Bounded physical planning calls `diskPhysicalAdmission` in the SQL executor.

- It shares the original SORT, aggregate, JOIN, set, ASOF, window and recursive admission kernels with allocation.
- The returned minimum is an admission limit, not a prediction of payload size. The final assigned
  budget must pass the same call before acquisition.
- SORT acquisition uses the original `admitDiskSort` with its exact equality and prefix layout.
- Mixed aggregation reports descriptor copies and separate acquisition and receiver shells. The graph
  scheduler owns simultaneous lifetimes.
- Temporary-file metadata and one sequential I/O peak come from `tempPagesMemoryCredit` of the Store.
  Caller buffers, held pages, concurrent I/O and the issuer directory stay separate charges under the same Resources.

Capture and plan sizes:

- `finiteInputCaptureBytes` measures the original retained context, physical plan and LIMIT-stage
  slots before the graph allocates them.
- Allocation uses the same pipeline walk in construction mode. Measurement never evaluates LIMIT or source expressions.
- Prepared membership keys are visited through the original live Map slots of the SDK, without an
  allocation of a keys vector.
- This is logical admission credit, not an RSS measurement.
- Plan and expression size admission use the original definition codec in measurement mode.
  `physicalPlanSize` counts a checked Rel. `expressionSize` counts a checked expression vector.
  Their node, depth and byte limits also apply in write mode.
- Type credit uses `typeWireSize` of the Value owner, without an allocation of an encoded header.

### I/O receipts

Every physical receiver exposes cumulative `ioReceipts()` from its original temporary-file lineage.
This includes merge files that retired successfully, and closed children.

- A transfer gives the receiving owner the same receipts. It does not make another I/O ledger.
- A successful close clears retained plan, context and payload metadata before it releases its grant, and keeps the counters.
- A failed cleanup keeps the original owners for a retry.
- Counts give actual File API transfers and operations. They do not give OS syscalls, device I/O or wire traffic.
- A failed or unobservable direction stays marked unknown. Credit estimates never replace measured I/O.
- The start of destructive cleanup also poisons later polls and source transfer, because a failed
  close can have cleared kernel state while its original files and grants stay live. Then only a
  cleanup retry under that issuer is permitted, also with no spare credit.

### File planning

A federated FILES source plans through `openFilesContext` under the statement ResourceContext and
FilePlanningLimits.

- CSV layout uses the record grammar of the SDK, with the original delimiter policy of the first 100 records.
- JSON schema inference walks validated value spans and keeps distinct fields, without a full nested JSON AST.
  Traversal admits its actual retained names and the current recursive stack. The declared match,
  path and depth limits stay hard work ceilings.
- The canonical compact parser of Parquet measures tree nodes, field IDs and binary spans without a
  retained tree, before that same parser decodes the footer.
- These are conservative logical credits, not allocator measurements.

### Cleanup retries

#### Context retirement

- Context retirement keeps its original pending arena and manifest grants if the cleanup I/O fails after the cache closes.
- That closed cache prevents more operations. A call of the same `StorageContext.close` under its
  original issuer continues the remaining cleanup without resource admission.
- Pending ownership and manifest references clear only after a successful release.
- A Native fixture injects a real refusal of a pending-file close, a repeated refusal, a refusal by a foreign issuer,
  and a retry with no free memory credit.

#### WAL writer close

- The shared writer revision of the WAL also keeps a failed file close in a closing state. Append and
  sync refuse. The original current revision can try the cleanup again.
- An obsolete alias cannot close that handle. Retirement is recorded only after the actual file close succeeds.
- Its fixture keeps the durable WAL cut across refusal and reopen.

#### Engine close

- `Engine.close` and `closeWithStorage` return an explicit I/O error when the WAL file refuses to
  close. They keep the original Engine for a cleanup retry.
- A closing WAL can be tried again only at a confirmed cut without unpublished writes. Sync errors of
  a stopped engine have precedence over close errors.
- Pending cleanup separately validates the original leases and the identity of the range that is not
  confirmed. Thus a scratch reset failure after a successful WAL fsync does not prevent the retirement
  of that poisoned arena.
- Normal pending reuse still refuses poisoned scratch.
- Pending-WAL fixtures cover both close APIs, an actual reset I/O failure, refusal of a stale generation, refusal of a
  later write, cleanup with zero free credit, and exact durable rows after reopen.

#### Restore cleanup

- Legacy restore errors carry `RestoreError.cleanup` when recovery acquired a WAL owner that refused to close.
- `RestoreError.close` tries the original Log and scanner again. It never opens a replacement file,
  and never runs the restore again.
- A successful cleanup clears the carrier. A refusal keeps it.
- The CLI reports the restore error, and reports the cleanup refusal separately.
- Restore with an actor context keeps the cleanup in the original `StorageContext`.
- The backup fixture injects a real close refusal after recovery, and checks a retry by the same owner without another file open.
- This covers recovery open and retirement. Copy and directory-discovery failures are separate paths. This carrier does
  not give them a general physical cleanup guarantee.

#### Opening cleanup

Open failures keep the original WAL cleanup owner separately from the pure `LogError` and `DbError`.

- Legacy Log factories return `LogOpeningError`. Legacy Engine open and inspect factories return `OpeningError`.
- Their semantic cause does not change. Callers try `.close()` again on that same error until its physical carrier retires.
- Openings with context transfer it into the `StorageContext`, with a cleanup slot of the original
  issuer admitted before the WAL is acquired.
- `retryOpeningCleanup()` retires the handle and the admission before reuse. A foreign issuer or a refused close keeps both.
- Verification reports and salvage errors keep the same carrier. They do not change it to text.
- Its fixture covers retries of the original handle, refusal by a foreign issuer, cleanup with zero free credit, and exact recovery of
  scanner admission.
- This contract covers WAL opening owners. Other direct physical opens still require their own original cleanup guarantees.
- Opening cleanup retires separately admitted scanner windows, also when a Log alias has already
  closed the shared File. A zero writer revision certifies only the retirement of the File. It cannot
  erase the original resource claims of the scanner. The recovery fixture covers this alias order and
  exact admission recovery.

### Native JOIN fanout credit

The native JOIN fanout credit includes its retained input deltas, the touched State receipts, the
peeked pair and the unconsumed output certificate.

- Each transition checks their overlap against the original execution allowance. Enqueue also
  subtracts output that is already retained in the same page.
- `nativeJoinRetainedCredit` reports this conservative logical admission. It is not measured RSS.
- Intermediate result pages keep a target that is not published. The coordinator refuses that state
  before it encodes or admits a checkpoint. Only the completed certificate advances the result frontier.
- JOIN and publication fixtures cover these boundaries.