GizDB: one database for records, analytics and search

GizDB gives transactions, analytics, full-text search, vector search, and live data from other systems, in one engine written in Gizlang. It runs embedded in a process, or as a server that speaks the PostgreSQL, Redis and Kafka protocols, with the same format.

This page is the design as it is built. For each decision, it gives the reason and what the decision replaces.

• A part is described only when its code and a committed test exist.
• A performance statement occurs only with a measurement in bench/db/ (refer to the README there).
• The code is in db/. Its tests are the *.test.giz files and the suites in tests/db (bin/giz test db); each scenario of a fixture such as pg.giz is a test in pg.test.giz.

Contents

0. From the requirement and the physics
1. Seven primitives, one owner each
2. Log
3. Version and transactions
4. Segment: one storage format
5. Derived structures and watermarks
6. Plan: one algebra
7. Sources
8. Correctness
9. Text analysis
10. What GizDB does not do
11. Native versioned journal
12. Native storage resources

Status

0. From the requirement and the physics

A user needs these things:

• A write is durable, and visible to every later read.
• A read sees one consistent state.
• A question about the data gets an answer at the cost of the data that it must touch. For example: a record by key, a sum over a billion rows, or the documents that best agree with words or an image.
• Data from other systems is current, without a pipeline that someone must build.
• The loss of a disk, a machine or a region loses no committed write.
• The same model serves an embedded app, a server and a cluster.

These limits decide the design:

1. Seven primitives, one owner each

Every product above implements the same few things separately. GizDB implements each one time. All other features are combinations.

Where the code is

The files at the top of db/ are the public boundary (PACKAGES: access, data, query, typed, value and the others). The rest is in folders: db/core/ (log, version, segment, tables, catalog, backup), db/server/ (the server, its actors and metrics), db/sql/ (lex, parse, bind, sessions and statements) with db/sql/types/ (SQL types and their values), db/sql/functions/ (built-in functions and aggregates), db/sql/exec/ (plan, optimizer and executor) and db/sql/catalog/ (system catalogs and catalog objects), and one folder for each protocol or source (db/pg/, db/kv/, db/kafka/, db/ora/, db/ingest/, db/ml/).

One engine, several interfaces

SQL (the PostgreSQL wire and the Gizlang API) and the Redis protocol are front ends of the same engine. They are not two databases.

• The Redis key space is the relations _giz_kv (one row a key), _giz_kv_items, _giz_kv_ordered, _giz_kv_ttl and _giz_kv_size (db/kv/README.md). SQL reads and writes the same rows, in the same transactions and snapshots.
• One copy of the data, one log contract and one recovery. The physical paths differ by work: a SET is a key mutation on a hash primary, a shipped autocommit UPDATE a key mutation on an ordered primary, and other SQL uses the ordered and columnar paths.
• With KV shards, each shard engine writes its keys alone, and the main engine reads and writes all shards' rows through parent relations in coordinated commits (db/kv/README.md).

Three engine mechanisms give key-value speed. OLTP needs each of them also:

1. The newest version of a row is updated in place when no open snapshot is older than it (HyPer). An older version is kept only for a snapshot that needs it. Thus a hot counter does not collect versions.
2. A container change is logged as its operation (HSET k f v) and replayed deterministically. It is not logged as the full value.
3. An ordered index serves ranges.

The rule of orthogonality

A new capability adds an operator to Plan or a function to Derived. It never adds a storage format, a log, a transaction mechanism or a permission path. A text search, a vector search and a sum are three plans over the same versions.

2. Log

Built: db/core/log.giz.

Format

• A position in the log is an LSN: the byte offset of a record in the full history of the log. Thus it only increases.
• The log is a series of files, each named by its first LSN.
• A record is [length u32][crc u32][body]. The crc is the CRC-32 (gizlang:compress) of the length bytes and the body, masked as LevelDB masks it. Thus a tail filled with zeros, or an embedded frame, never verifies. Length 0 is never written.
• A body is one commit: its timestamp and its row operations for each table. Its maximum is 4 GiB (the range of the length). A larger commit fails with 54000.
• The operations of a commit are kept encoded as the log writes them, and framed where they are. The checksum of the parts is joined arithmetically. Thus the log buffer holds no second copy of a large commit.

Recovery

Recovery reads records from the LSN of the checkpoint.

• The log file grows by 4 MiB of written zeros at a time (File.writeAhead). An append then overwrites zeros, and its sync flushes only data: a 4 KiB synced write on backup takes 1.8 ms into written zeros and 6.2 ms when it extends the file. A zero rest of a file is where its records end.
• A record that fails is a torn tail only when no valid record follows it in the last file. It is then cut off.
• A failed record with valid records after it, or a gap between files, is corruption. It stops the open. A committed record is never dropped silently.

The candidate recovery API returns metadata (Opened.records/from, a cursor and FrameRange), not a vector of bodies.

• CRC verification, the header search in a damaged tail, and the canonical file-backed Reader windows read a maximum of 1 MiB for each physical read.
• Short reads continue. An early end of file, compared with the sampled file size, is an I/O error. It cannot authorize a truncation of the tail.
• The bytewise search keeps its header window. CRC candidates use independent bounded ranges.
• The Engine validates a complete commit and its Journal offset pairs in sequence. Then it applies them through the same mutation owner that live receive uses.
• A writable reopen still syncs before it publishes durability. An inspection never repairs or syncs.

Note: The allocations of Values and schemas for each operation, and the resident table and index state, are separate resource work. These windows do not prove fixed total RAM or a capacity of terabytes.

Commits that answer before their sync

By default a commit answers after the sync of its log record. gizdb --synchronous-commit off answers once the record is in the log file, before its sync, as PostgreSQL's synchronous_commit = off does (TechEmpower configures PostgreSQL so). A crash of gizdb loses no answered commit. A crash of the host can lose the last ones. A read sees a commit before its log write. A commit that a standby must confirm (--sync-slot) still waits for its sync and receipt. On TechEmpower /updates (20 rows a request) it raised GizDB from 5,485 to 6,442 req/s, p99 from 171 to 65 ms (a build host, 10/17).

Durability failures stop the engine

A failed write or sync is not tried again. After a failed fsync, the kernel can drop the dirty pages and report them as clean (the fsyncgate of PostgreSQL). Thus the engine refuses more work until it is opened again and has recovered from the log.

Bounded raw replication chunks

• Log.chunk and chunkReaderWithResources share the lookup of retained files and the canonical range Reader.
• A request stays within its byte credit and one physical file, also inside a large frame.
• The scoped sender keeps its Reader until the outgoing bytes are released.
• Ordered rotation cuts use binary search. Invalid signed frame offsets are refused before a read.
• A transport cut alone does not certify a verified, applied or durable commit. The receiver must establish those boundaries through the same WAL checksum and Engine publication owner.

Bounded base backup and PITR copy

Restore:

• restoreWithResources parses the bounded UTF-8 completion label while its buffers stay admitted. Its replay coordinate must agree with the checksummed MANIFEST. It verifies a complete WAL frame boundary before it writes the target. Invalid metadata cannot move or bypass the PITR cut.
• It uses the same restore owner as restore, the ReadResources of the caller, and the WAL FrameScanner for checksum validation.
• The exact-range copier of fs/disk.giz copies file bodies. No backup segment or complete WAL body is collected in memory.
• A short read or write continues. An I/O failure stays an error. Every owned handle and window is released before the return.
• A restore still requires a destination that is empty at the start and exclusive. Partial files of a failed restore are never reported as recovered.
• Replay coordinates in the label that are missing, malformed, too large, negative or duplicated refuse the restore before any destination file is copied. They never become position zero.

Backup:

• Engine.backup uses the same copier with a 64 KiB window. backupWithResources uses the Store admission of the caller.
• backupWithContext checkpoints through the actor's storage first, then copies the segment files and the logs; evicted segments are copied from the same immutable files.
• Segment and log bodies are complete before the manifest is copied. BACKUP_LABEL certifies completion.
• The base-backup fixture refuses whole-file reads, and verifies independent totals of rows, counts and lengths after reopen. It repeats the copy with a 1 KiB window under an admission limit of 16 KiB.
• The fixtures for admission denial, partial writes and fsync errors require no completion label, returned credits, and unchanged query results on the source. A failed destination does not damage the source database.
• A fully published persisted checkpoint uses the same backup owner to copy its primary and end roots and its segment data and index files. A stored overlay that is not published refuses backup until its canonical checkpoint and replay with context finish.

Restore with context:

• restoreWithContext copies through the same restore owner and opens through openWithContext. It returns the canonical Engine under the actor context of the caller.
• The cold-root fixture checks the visibility of updates and deletes, and historical point lookup, without a rebuild of resident rows or primary maps.
• restoreRecoveryWithContext gives an explicit canonical resource policy and calls openRecoveryWithContext through the same format dispatcher. A cold archived tail then uses the writable WAL replay and the MVCC publication owner. A replay that is not supported is refused. It does not change to resident hydration.

WAL archive:

• Closed WAL archival uses the atomic range publisher of the Store. It does not read a full file into memory.
• archiveLogsWithResources admits its copy window on the same ledger of the caller. Retention moves only after a successful publication and cleanup of handles and windows.
• archiveNowWithContext first flushes the pending WAL of the actor through syncWithStorage. Then it uses the same publisher and ledger. The path without context cannot flush pending pages that an actor owns.
• The archive fixtures check bounded byte-for-byte copy, no work for a cut that is already archived, and that the previous complete file and retention cut stay after an admission refusal or a failure before publication.
• A failed directory sync after a rename is uncertain durability. The new complete file can exist, but the retention cut does not move.

The point-in-time SQL fixture adds a source that refuses whole-file reads, and a payload of 256 KiB under a copy admission limit of 16 KiB. This tests the geometry of the copy, not the capacity of the full database.

Note: Directory and log metadata still increase with the number of files. The final legacy openOn recovery keeps resident tables and indexes. These APIs do not certify bounded total RSS, native tables larger than RAM, or a completed implementation of mutable online backup.

Group commit and checkpoint

• Group commit. Commits that arrive while a sync runs share the next sync. A transaction is acknowledged only after the sync that covers it.

• Checkpoint.

  1. Start a new log file at the current LSN.
  2. Write the delta as segment files.
  3. Replace the manifest (staged file, sync, rename, directory sync). The manifest names the segments, their deletion state and that LSN.
  4. Only then, remove the older log files.

  A crash at any point replays from the LSN in the manifest that survived. Replay is idempotent by LSN.

• Why not a B-tree with a WAL (SQLite, PostgreSQL). Pages that are updated in place need a WAL also, and write each change two times. A log plus immutable segments writes one time to each.

• Replication sends these same records. A change feed is a reader of them.

Standby

Built: db/pg/replica.giz. Start it with gizdb --primary host:port and the password of the primary.

Copy and follow:

1. An empty directory is copied from a checkpoint of the primary: its manifest and the segment files that it names. The primary keeps them until the copy is done. The manifest is written last, so an interrupted copy starts again.
2. Then the log of the primary streams from that LSN. It is applied as recovery applies it (Engine.replay, which both use) and appended. Thus the log of the standby is the log of the primary, byte for byte, and a restart continues where its log ends.

Slots:

• The standby is an ordinary client of the primary (REPLICATION statements, superusers only).
• A request with nothing to send waits for the next commit, as changes() does.
• A request carries the durable end of the standby to its slot. The primary keeps its log from the slot across checkpoints and restarts (the file SLOTS, as the replication slots of PostgreSQL). Thus a standby that restarts or was away continues.
• REPLICATION SLOTS lists the slots. REPLICATION DROP n frees the log that a slot held for a standby that will not return.

Reads, writes and promotion:

• Reads on the standby see each commit shortly after the primary acknowledges it (asynchronous, RPO > 0).
• Writes fail with 25006 until a superuser runs SELECT pg_promote(). This must be the only, unconditional autocommit command.
• Ordinary users receive 42501. Transaction blocks receive 25001. Conditional, subquery and routine calls are not supported, so they cause no side effect at bind time.
• The authoritative engine forks its branch, and atomically replaces and syncs REPLICATION_HISTORY, before it acknowledges promotion or permits writes.
• A restart with the original --primary command keeps the promoted role.
• An uncertain sync of the role file stops follow and writes, and requires a restart of the process. The restart reads the durable role again. Ordinary automatic storage retries cannot continue the old role against a new branch file.
• pg_is_in_recovery() tells which role the authoritative server has.

Branch identity:

• Each live REPLICATION LOG lsn durable slot 'history' request checks its expected random branch identity before it moves a slot or returns bytes.
• Promotion assigns a new identity. Two conflicting histories at an equal LSN are refused, also on a client that is already open.
• REPLICATION STATUS reports the branch, the durable LSN (NULL when storage is uncertain), the committed timestamp, the recovery role and the latest replication or storage error.
• Lag is measured by a comparison of the durable positions of a primary and its standby. It is not inferred from row counts.

Designated synchronous receipt

This is opt-in.

1. Provision a standby base asynchronously.
2. Restart its primary with --sync-slot ID --sync-timeout 5s.

Rules:

• Actual commit replies wait for the local WAL sync, and for a new LOG acknowledgment with a matching history of the fsynced receive of the designated standby.
• BASE and the persisted SLOTS retention positions never certify receipt. Reopen and promotion clear live confirmations.
• STATUS appends sync_slot, confirmed_lsn and sync_waiters.
• Missing slots, invalid options, and a drop of the selected slot are refused.
• On timeout, FATAL08007 states that the transaction is committed and durable locally, but that remote receipt is not confirmed. Then the server disconnects. Examine the state before you try again. Do not assume a rollback.
• Writable recovery syncs the recovered tail before it publishes a durable position. Failed sync or replay reopens close their log handles. Inspection stays read-only.

Tests: local process tests with four loops cover absence, stale and reopened confirmations, Flush and Sync, the exact acknowledged rows after a SIGKILL of the primary, a standby restart with the upstream offline, and durable promotion.

Warning: This trusts the superuser receipt of one designated standby, without consensus or automatic failover. The new writes of a promoted node are asynchronous by default. Provision and explicitly select its new downstream before you claim synchronous durability for them.

Divergence and rejoin

• A standby history that is missing or malformed fails closed. Standbys from before the history file require a new base.
• A divergent old primary, and a downstream standby whose upstream was promoted, rejoin through an empty directory. The old directory stays for reconciliation.
• There is no rewind and no automatic adoption of a branch. Fence the old primary before promotion. After a permanent promotion, explicitly drop its obsolete upstream slot when that primary is reachable.
• Segment copies keep a maximum of one decoded chunk of 4 MiB. They do not buffer a complete segment.
• An interrupted base publishes no MANIFEST. A retry replaces partial files, and releases its recorded abandoned slot only on the same branch. A crash before the slot is recorded can leave an orphan. REPLICATION SLOTS shows it for explicit cleanup.

Replication tests and endurance

Local process regressions (tests/db/replication) exercise actual SIGKILL, live catch-up, primary and replica restart, divergence at an equal LSN, promotion authority and persistence, restart of a partial copy, rejoin with a new base, and exact ordered rows with concurrent ingest and analytical readers.

bench/db/reliability/replica.giz [milliseconds] runs one continuous disposable cluster, for fifteen minutes by default. It has a fixed rolling dataset of 16,384 rows, repeated crashes, generation-aware row checks, bounded latency histograms, durable lag and resource samples.

The measurements are in bench/db/README.md. The original timeout is not explained.

Startup log replay builds the optimizer statistics one time, from the final recovered state. Queryable commits and live replication keep ordinary resampling. A bounded probe of a retained copy verified exact cells through replay, checkpoint and reopen. It counted 458 startup scans that became one, and confirmed checkpoint reclamation (details in bench/db/README.md).

Not supported: automatic failover based on consensus, acceptance across hosts and after power loss, and sustained synchronous ingest performance. The local tests of designated receipt do not prove these.

3. Version and transactions

Built (db/core/engine.giz, db/core/table.giz), except hybrid logical clocks, which wait for the cluster.

• A row version is visible to a snapshot s when begin <= s < end. end is infinity while the version is current.
• An update or delete sets the end of the current version to the commit timestamp. An update also appends a new version.
• Deletion state is a column of end timestamps for each segment row. It is persisted at checkpoint and built again from the log after the checkpoint. Thus segment files never change.

Writers and readers

• One writer applies commits in series. Readers read snapshots and never wait.
• Transactions are optimistic (the design of FoundationDB).
  • A transaction reads at a snapshot and buffers its writes. Its own reads see them. The buffer is indexed by key and unique value, so a read back costs one probe at any size.
  • It records what it read: keys, and the tables that it scanned.
  • Commit is one serial step of the writer. If a commit after the snapshot wrote something that the transaction read, the commit fails with a conflict, and the caller tries again. If not, its writes are applied with the next timestamp and logged.
  • This is serializable. It holds no locks while the client thinks. It agrees with how the Giz app server works: one transaction for each HTTP request, with awaits between its statements, committed at the end (platform/server/src/common/api.interceptor.ts).
  • A transaction given as a function runs inside the writer and cannot conflict.

Isolation levels

Tx.isolation (tests/db/isolation.giz) gives the isolation levels that a client of each engine expects.

• BEGIN ISOLATION LEVEL, SET TRANSACTION and SET SESSION CHARACTERISTICS select the level.
• READ COMMITTED reads a new snapshot at each statement. REPEATABLE READ (and SNAPSHOT of SQL Server) keeps one snapshot.
• Both validate at commit only the rows that they write, the unique entries that they use, and the foreign-key checks. Thus readers never cause conflicts. A commit with no commit after its snapshot looks none of them up: none can have changed (Engine.validateReadsIn).
• The changed rows of a statement, and SELECT ... FOR UPDATE, take row locks (Engine.lockRows, NOWAIT, SKIP LOCKED).
  • A second writer of the row waits. Then it runs its statement again on the new snapshot (no lost update).
  • A READ COMMITTED write, and SELECT ... FOR UPDATE, read the newest commit, durable or not: a commit frees its row locks before its sync (early lock release). The answer that ends the work waits for the sync of each commit it read: the statement's own when it runs alone, else its transaction's COMMIT or ROLLBACK (Client.seen).
  • REPEATABLE READ fails with 40001. A cycle of waits fails with 40P01.
• A statement alone checks locks only while a transaction holds one.

Foreign keys

Foreign keys are true at each statement (Fk in db/sql/catalog/keys.giz).

• A written child row must name a parent row that exists (23503).
• A delete or change of a referenced parent row restricts, cascades, or sets null or the default, inside the same transaction.
• The checks are reads. Thus a child committed against a parent that a different transaction deletes conflicts at the later commit. It does not become an orphan.
• Not supported: DEFERRABLE keys. They are checked at each statement, not at COMMIT.

Named locks

Named locks wait (as pg_advisory_xact_lock of PostgreSQL).

• The second taker waits until the holder ends. A wait that closes a cycle fails with 40P01.
• When a transaction gets a lock, it validates what it read until then, and moves its snapshot to the latest commit. Thus its next work sees the writes of the previous holder, and does not conflict with them.
• Fifty concurrent transactions each lock their own chat, read it through an index, and append to one feed under a shared lock. All commit without a retry. tests/db/pg.giz holds the case with two transactions.

Timestamps and old versions

• Timestamps are hybrid logical clocks (physical milliseconds and a counter). Thus the logs of one node need no format change for the cluster.
• Old versions are kept until no open snapshot needs them. Then compaction drops them.

4. Segment: one storage format

Built (db/core/segment.giz, db/core/table.giz), except the quantized vector copy. A table is a list of segments and one delta.

Delta and segments

• The delta holds recent versions as rows. Appends and point reads are cheap.
• A segment holds a maximum of a fixed number of rows, as columns, in the order in which the delta flushed them.
• Each column is divided into blocks of 4096 values, with a zone map (min, max, nulls) for each block.
• A scan reads only the blocks that a column op constant condition can match:
  • integers exactly;
  • floats, without trust in a bound that NaN or rounding could cross;
  • text by its position in the sorted dictionary;
  • decimals of one scale by their coefficients.
  • <> skips a block that holds only its constant (integers and text). pg_stat_gizdb.zone_skipped_rows counts the rows that the zone maps skip.
  • A cold text column against '' reads only the lengths of its dictionary, not its strings.
  • When one condition rules out every block of a part, the others are not checked.
• min and max of an integer or date column over a whole table, with no condition, come from the zone maps alone when every part is a whole segment whose rows are all visible.

A segment file (version 3, GIZSEG03) is a sealed head and one sealed block for each column. The head holds the row count, the begin stamps, and for each column the offset and length of its block and its zone maps. Thus an open reads only the heads, and a query reads only the blocks of the columns that it names (ColdHome.column).

Encodings in a block:

A version 1 file (GIZSEG01, one sealed blob of every column) still opens, whole and resident. It is not in the files that a checkpoint names, so the next checkpoint (or eviction) writes it again as version 3.

A primary-key lookup goes to the delta, then to the key index of the segments (§5), newest first.

Compaction

Compaction merges small segments into large segments, and drops versions that no snapshot needs. It builds the derived structures of the result again. It runs inside a checkpoint, in three steps (db/core/engine.giz):

1. The writer plans it: a new log file, the deltas flushed, and the segments and ends of the tables taken as values.
2. A thread of its own computes it: merges, segment encodings and the manifest. runCheckpoint reads only the plan. Commits continue meanwhile.
3. The writer finishes it: files, manifest, log trim.

• The merged segment then replaces its inputs in memory, with the ends written meanwhile and the index entries moved (Table.install). It does not do this when a DDL came between.
• A plan older than the manifest on disk is dropped (tests/db/checkpoint.giz).
• The flush of a delta into a segment still runs on the writer.

This gives point reads at the cost of an index probe, and scans at column speed, from one copy.

Why not two copies, rows for OLTP and columns for OLAP. Two copies double the writes and need a replication path between them. The delta is the row copy. It is bounded and lives for a short time.

5. Derived structures and watermarks

A derived structure is a pure function of a segment (and of the delta, incrementally). It is stored next to the segment and built again by compaction. The key index, the zone maps and the inverted index (db/core/inverted.giz) and the vector index (db/core/vector.giz) are Built.

Each derived structure records its watermark: the commit that it shows.

• A plan that uses a structure reads the commits after its watermark from the delta, or waits. For the structures above, the read from the delta is always possible, because the delta is searched exactly.
• Results never claim a freshness that they do not have.
• Asynchronous structures (embeddings that a model computes, materialized views over sources) report their watermark with the result.

Live materialized views

CREATE LIVE MATERIALIZED VIEW v AS SELECT keys, aggregates FROM t [WHERE ...] GROUP BY keys [HAVING ...] makes a table of the query's rows. Each write to t changes v in the same transaction (Conn.maintainLive in db/sql/liveview.giz). Its watermark is always the reader's snapshot: a commit shows t and v together, and a rollback undoes both.

• The query reads one table. Each GROUP BY expression is also a select item. Without GROUP BY, the view is one row.
• A statement that only inserts adds the partial aggregates of its new rows to their groups, when every other item is count, sum, min or max and there is no HAVING.
• Any other write computes again the groups that its old and new rows touch. Past 1000 groups, or without GROUP BY, it computes the whole view again.
• The write runs on the row path: a table with a live view takes no bulk load or bulk rewrite.
• TRUNCATE of t computes the view again. REFRESH MATERIALIZED VIEW v does too.
• Only maintenance writes v (42809). DROP TABLE t needs CASCADE, which drops v. DROP MATERIALIZED VIEW v drops it.
• Two transactions that write rows of the same group wait for each other on the view's row, as under READ COMMITTED.
• A plain CREATE MATERIALIZED VIEW is a view: it is always fresh, and REFRESH does nothing.

6. Plan: one algebra

Built (db/sql/): parse (the spellings of SQLite, PostgreSQL, MySQL, SQL Server and Oracle), bind, optimize, execute, and the session that runs statements in transactions.

Bind

bind.giz resolves names, folds aggregates and windows into their operators, and changes subqueries into plans.

• An uncorrelated subquery runs one time.
• A correlated subquery becomes APPLY, with a cache by outer values. It goes under the aggregate when the select groups, and under the sort when ORDER BY reads it.
• A subquery written two times is one plan. Thus an item that repeats a GROUP BY key reads the key.
• Views and CTEs expand in place.
• WITH RECURSIVE binds its first SELECT as the anchor, and the remainder as a step that reads the rows of the last turn (RECURSE over a WORK leaf). It runs until a turn adds no rows. UNION drops rows that it already produced. Each turn checks statement_timeout.

Optimize

optimize.giz does these steps:

• It pushes predicates through projections, joins and aggregate keys.
• It extracts equi-join keys. This includes a conjunct that every branch of an OR holds.
• It unnests correlated scalar and EXISTS subqueries into joins.
• It orders inner-join chains by the lowest estimated cost: dynamic programming over subsets, and greedy after DP_INPUTS. Each step hashes the smaller side.
• It changes equality on a primary key or a full index into a lookup. The value may be a parameter or a library function of constants and parameters that is not volatile (id = MOD($1, 1000) + 1).

Estimates:

• A join on the unique key of a side is estimated as a foreign key: the rows of the other side times the fraction of that table that is left.
• A key of several columns has a maximum of as many values as its side has rows.
• A join of few rows with a grouped aggregate on its group keys first semi-joins the input of the aggregate with their keys. Thus the average for each part in TPC-H Q17 reads only the joined parts.

Index ranges:

• A selective comparison on a btree index column (after equality on its leading columns), or ORDER BY its columns with LIMIT, becomes a walk of the key range of the index (RANGE, forward or descending).
• Every condition is still checked on the rows that the walk returns.
• The walk is recorded as a read of its key range.

EXPLAIN prints the plan with row estimates.

Execution receipts and budgets

Finalized invocation profile slots accept measured execution receipts through Exec.recordOperator:

• emitted rows and the invocation count;
• elapsed milliseconds (optional);
• actual work rows, kernel calls and polls;
• successful spill read and write bytes (optional);
• cancellation events.

Rules:

• Receipts are deltas that the execution owner publishes one time. A record never executes a plan.
• Optional counters keep their sample coverage. Thus missing or partial instrumentation is not a measured zero or a complete operator total.
• Spill byte receipts give successful physical transfer lengths, with framing. They do not give temporary capacity, payload estimates or process I/O.
• The invocation owner serializes trace mutation.
• Stored-graph reports and their physical I/O instrumentation need their receiver integration. This API alone does not prove them.

Exec.partitionExecutionBudget divides one state allowance among the declared graph owners and their exact live phases.

• Connected prepared SORT sources and their receiver receive the same budget. They keep the exact admission checks of the source.
• Simultaneous minimum grants must fit before spare capacity is shared.
• Disjoint phases can use an allowance again only after their resource owners close.
• Row, output and work limits for each poll stay the configured limits.
• The fixed planning credit and the declared overlap that is not state are included in the computed peak.
• This peak is a conservative grant limit. It is not observed RSS or elapsed performance.
• Runtime enforcement requires the execution fence of the same issuer in ResourceContext. A phase schedule of a caller that is not complete is not proof of bounded execution.

Execute

exec.giz processes morsels of 16384 rows as columns.

Operators:

• Pipelines (scan, filter, project, probe) run in parallel by morsel (Task.parallel).
• Hash join builds a table divided into 64 partitions in parallel: inner, left, full, semi, and anti with NOT IN semantics.
  • It uses only the columns read above it, its keys and its condition.
  • It filters candidate pairs 65,536 at a time. Several integer keys are packed into one exact number.
  • A semi or anti join without a residual condition builds only key membership. It permits duplicate integer keys, and needs no concatenated payloads and candidate pairs.
  • A selective semi join moves below plain column projections and repeatable inner joins when all its keys read one input. It keeps the multiplicity of that input. Outer joins and computed projections stop it.
• Hash aggregate pre-aggregates in each task and merges by partition. Counts and sums of integers, floats and decimals of one scale use arrays for each group.
  • Typed groups (typedAggregate) take a GROUP BY whose keys are integers, booleans, dates, times or text, and whose aggregates count, sum, average, take an extreme, or take a DISTINCT of one integer column. A key is a 64-bit word: a text key is its string's hash, which a batch computes once for each dictionary entry its rows use; a group keeps its strings, which decide when the words agree, and the merge moves them with the group (no map of strings). The groups are one open-addressing table of words, whose slots keep 23 bits of the group's hash, and each aggregate keeps one array of state for each group (DuckDB's aggregate_hashtable.cpp). A sum that passes an i64 keeps that group in the row loop's exact state. A key that is another key plus, minus or times constants (ClientIP - 1), or equal to an earlier key, is not hashed: the finish computes it from that key's column for every group (typedKeys).
  • A group costs its key words, a hash, two slots and 9 to 41 bytes for each aggregate: an aggregate makes only the arrays of the values it meets (TypedAgg.made), so a NUMERIC sum takes 17. The tasks hold at most one entry for each input row, and the merge holds each group once more.
  • A task's table that reaches 16,384 groups (TYPED_LOCAL_GROUPS) doubles its limit when its groups will reduce the task's rows (typedGrows): the rows it took to fill give the keys' distinct values as if drawn at random (typedDistinct), and from them the groups of the task's rows. A table past three quarters of a core's share of the last-level cache (gizlang:machine cacheShare: 4 MB on a Ryzen 5900X, 32 MB for each 6 cores; 4 MB when unknown) grows only for eight rows a group or more. Otherwise the table goes to its partitions (a piece). When its groups held fewer than two rows each, the task's later rows go to the partitions as they are, their keys and arguments only (TypedGroups.spread), and each partition's merge folds them (spreadRel, foldSpread). A task joins the rows of its morsels into batches of 65,536 rows before it splits them (SPREAD_ROWS): split per morsel, TPC-H Q15's 3,000 morsels made half a million columns of a few dozen values, and freeing them was half the query. pg_stat_gizdb.spread_rows counts these rows. The probe hashes 256 rows at a time and reads their slots ahead.
  • A GROUP BY on one integer key meets rows stored in key order (TPC-H Q18's l_orderkey): a task's groups are then made in ascending key order by appending, with no hash and no slot (TypedGroups.ordered; a batch out of order puts the groups in the table and goes on as above). The tasks read key ranges of their own, so only a key that crosses two tasks can be in two of them: orderedJoin merges those groups and the groups need no partition, no merge table and no copy. A task's ordered groups may hold the state less the merge's half. When the tasks' keys do not ascend from one to the next, the groups are partitioned as above (more passes when a copy would not fit). pg_stat_gizdb.ordered_groups counts the groups finished this way.
  • The typed groups check their bytes per morsel against the statement's state (Ctx.stateBytes). A task that spreads foresees its bytes at its end, and may hold the state less the groups its merge will make. Past it they first take, once, what is left of the memory no credit counts (Engine.grant: a quarter of the process's, shared by every loop) and give it back at their end. Past that too they read their input again in passes: each pass folds only the rows whose keys hash to it (a text key by its string), so it holds a share of the groups, and the passes are enough for each to hold half the state, as DuckDB's radix partitions do without the spill (the input is columns, read again with no write). A top-k keeps its first groups of each pass (pg_stat_gizdb.group_passes counts the passes past the first). Past 64 passes they fail with HASH_OVERFLOW, and the statement runs again on the estimates, which sort when the groups do not fit. Beside the state, the run holds the columns the cells keep (the tables budget), one window read past them (at most half that budget, resolveParts) and about 1.3 MB a task (its morsel's rows, their hashes and its table). None of these grows with the input or the groups: groupspill at a 4 MiB state needs a 56 MB process limit on one core for 1M rows and 72 MB for 4M rows.
• Stored statements run a plan of scans, filters, projections, hash joins, aggregates, sorts and LIMIT whole on this executor (a join only below an aggregate or a top-k sort, whose rows stream into it; a build past half the statement's state plans again with HASH_OVERFLOW) (a STORED_RESIDENT node), when its estimated state fits half of the statement's. In the first (optimistic) build, the state of typed groups counts as zero, because they check it as they grow. EXPLAIN ANALYZE shows the rows of each node of that plan.
• Sort orders typed keys (raw integers, decimal coefficients, text by dictionary rank) in parallel chunks merged in pairs. Under LIMIT, it keeps a top-k cut. A top-k whose first key is a count of the typed groups below it (GROUP BY ... ORDER BY count(*) DESC LIMIT 10) has each partition of groups finish only its first groups by that count, and every group tied with the last of them when more keys follow (aggregateTop): the other groups' aggregates (an AVG's NUMERIC each) are never computed. A LIMIT with no ORDER BY over typed groups (through projections, no HAVING) finishes any LIMIT groups of each partition, and its passes stop once it has them (aggregateAny).
• A scan's first filter reads its own columns for every row of a morsel, and the other columns only for the rows it keeps.
• A filter drops the columns that only it reads.
• With no GROUP BY, sums of one integer x plus constants (sum(x + 1), sum(x - 3), sum(x)) share one sum, count, minimum and maximum of x, and each answer is sum(x) + k * count(x), as DuckDB's SumRewriter (offsetSums). When x + k can pass its width for x's least or greatest value, or a total passes an i64, the sums run as written and fail or widen as they do.
• Compiled scans (compiled.giz): a scan's filter, and with no GROUP BY its aggregates (count, sum, avg, min and max) too, become one Gizlang function over a part's stored columns: one loop that reads each row once (Umbra's pipelines). Its text is generated from the plan (integer, date, time, NUMERIC and text-code comparisons, integer, NUMERIC and double arithmetic, AND, OR, NOT, IS NULL, IN), with its constant parts evaluated once (vm.giz folded). NUMERIC columns of one scale of up to 18 digits are their coefficients; the Oracle dialect, whose NUMBER trims each value, keeps NUMERIC arithmetic on the interpreter. gizlang:functions compiles the text once per sha256 in the background, at -O2, into GIZLANG_CACHE/functions. Until the unit is loaded the statement runs on the interpreter, which stays the reference: a part whose columns are of other kinds, or a run whose integer arithmetic passes its width, runs there too and fails as it does. Constants are the run's params, so a unit serves its shape with any constants. An installed toolchain's bin/gizdb runs the server with its version's CLI as GIZLANG_CLI (launcher.giz databaseEnvironment), so it compiles each new shape as a build does. A program with no GIZLANG_CLI (a server-only release, which holds no compiler) compiles none: it links the units its warm-up met (build --warmup, with bench/db/engine/warm.giz running the TPC-H shapes), and any other shape runs on the interpreter's bit filters (pred.giz compareBits: integer, date, time, NUMERIC, float and text columns against constants). A GROUP BY of distinct text columns over a scan compiles too (compiled.giz groupKernel): the filter, the arguments' arithmetic and the aggregates in one loop, a group a slot of its keys' dictionary codes (at most 4,096 slots, as DuckDB's perfect hash aggregate), whose run merges into the typed groups (TypedGroups.foldSlots). pg_stat_gizdb.group_kernel_runs counts these runs. The filters and the joins that follow a stored scan compile as one loop too (compiled.giz chainKernel, a pipeline's Source.chain): a row is read once, probes each inner, semi or anti join of one integer key (or a packed key of several) through its rank index or its key tables, and keeps the build rows in registers; the columns of a build that a later key, condition or residual reads are loaded from the build row. Only the rows that pass every link are written, each as its scan row and the build row of each inner join, and the interpreter gathers the columns of those rows alone and goes on with the steps after the last link the loop covers. How a build finds its keys (rank index, Bloom filter, tables) is a param, so one unit serves a join shape at any scale. TPC-H Q5 probed 60M lineitem rows through four joins on the interpreter, each a pass over the rows and a gather of every column. pg_stat_gizdb.chain_kernels counts the pipelines that ran with a loaded loop.
• A scan is bound by memory, not by the cores (TPC-H Q6 reads four columns of 60M rows: 1.9 GB as 64-bit values, 36 GB/s on the bench host). A kernel reads a scan's integer and NUMERIC columns as lanes of 8, 16 or 32 bits, the narrowest that the zone maps of every part keep their values within (Part.laneBits, Segment.laneBits; a cold segment only, which a segment a checkpoint wrote becomes at once, its columns kept in its cells, as on reopen: Engine.evictCold): a 16-bit lane reads a quarter of the 64-bit bytes, an 8-bit one an eighth (a flag, a small code). A lane is a copy of the column built once on every core before the morsels run (ColdHome.keepLane, ColdHome.keepShort, ColdHome.keepTiny), kept in the same cells as the columns, counted in the same budget and released with them. The kernel's shape names which columns it reads as lanes, so one unit serves a scan at any scale; a part whose lane is missing runs on the interpreter. A core running a morsel only reads lanes, which Compiled.lanes carries. pg_stat_gizdb.lane_kernels counts the pipelines that ran with lanes.
• LIKE of a text column against a constant pattern without _, an escape or case folding is a kernel test too: before the morsels run, each part's dictionary is matched once in dictionary order, on every core, in tasks of 65,536 entries (pipeline.giz dictsOf), and a row then reads one bit by its code (Compiled.dicts). A row-by-row match read a string object at a random place for each row: 200 ns a row on TPC-H's 2M part names (Q9, Q20) and on its 15M order comments (Q13). A part whose morsels read under a quarter of its rows (a top-N's zone cut) matches only the entries those rows hold.
• An integer IN list and IN a subquery's integers run over the batch as a binary search (vm.giz memberKernel); COUNT(DISTINCT integer) with keys is one operator (typed groups keep (group, value) pairs: a task in a set of 65,536 slots that it empties into a list when full, a partition in one set sized for all the pairs it takes); COUNT(DISTINCT integer) without keys sends each key, but a repeat of the row before, to a hash partition, which counts its keys in a set (exec.giz distinctKeys; a text key sends the strings of each part's codes; pg_stat_gizdb.distinct_keys); a top of count() by one integer or text key (ORDER BY count() LIMIT k, no other aggregate) sums each key's runs in such a partition's table and keeps each partition's first k (exec.giz topCounts; a text key's tasks count each part's dictionary codes and send each code met to its string's partition; pg_stat_gizdb.top_counts); the COUNT of the other side of a LEFT join grouped by the preserved side is the sum of counts a join key (optimize.giz eagerCounts). A text GROUP BY key that its table's primary key fixes (TPC-H Q10's c_name, c_address, c_phone, c_comment) is not grouped: under ORDER BY ... LIMIT the groups by the other keys give their top rows, and a join on the primary key reads the text for those rows only (optimize.giz lateKeys). A GROUP BY key that the other keys fix, a constant or an expression of integer key columns (ClickBench's GROUP BY 1, URL and ClientIP - 1 beside ClientIP), is not grouped either: a projection above the aggregate computes it for each group (optimize.giz impliedKeys). A WITH query of aggregates or joins that the statement reads twice or more runs once, before its readers, as a subquery of kind MATERIAL; each reader scans its batches (bind.giz Binder.table, a WORK leaf of kind WORK_SUB): TPC-H Q15 computed its revenue twice.
• Window functions, set operations, LIMIT/OFFSET and generate_series complete the set.
• A transaction reads its own writes.

Window frames

Window aggregates keep one mergeable state for each row.

• A frame from the start of the partition (a running total) folds each row one time.
• A sliding frame merges O(log n) states of a segment tree in row order.

ROWS frames:

• They use signed preceding and following offsets, also an empty frame.
• SUM frames whose integer sign totals can overflow fold their rows in order. This keeps the intermediate-overflow errors of the aggregate, and does not refuse a safe selected frame.
• ROWS offsets are clamped before the addition. Thus the largest i64 bound cannot overflow.

RANGE frames:

• They support unbounded and current-row peer frames, and numeric distances over one ORDER BY expression. This includes ascending and descending order, NULL peers, and fractional decimal and float offsets.
• Each peer group shares a frame. Binary searches over its representatives find the boundaries by value.
• PostgreSQL integer and decimal thresholds stay exact. This includes values above 2^53, and sums larger than the stored range of 40 digits.
• SQLite uses its scalar arithmetic owner for REAL rounding and integer overflow at distance comparisons.
• Offsets are expressions without rows, evaluated one time for each window. NULL, negative, NaN and incompatible offset types are errors.
• Timestamp and interval distances, and scalar-subquery offsets, are not supported.

Named WINDOW definitions resolve within their own SELECT, with inherited partition and order clauses and explicit ROWS frames. Invalid references and inheritance are errors.

VALUES and numeric UNION arms use the numeric common-type conversion of CASE and COALESCE. Thus mixed integer and float, or integer and decimal, data keeps numeric order.

Runtime filters and ASOF joins

• An inner or semi hash join on one integer key gives its probe scan the range of the build keys as a condition (a runtime filter). Thus zone maps and Parquet statistics skip the data that no build key reaches.
• A hash join on integer keys over a range of at most 2^28 values, not sparser than a key in a thousand values (and a megavalue), is a rank index (Built.ranked): a bit for each value of the range, the keys before each 64-bit word, and the build rows in key order (order, with firsts when a key has several rows). A key's number is its rank (the bits before it, by popcount), so a test is a bit and a lookup is two reads, with no hash and no table; keys probed in order read all of them in order. TPC-H SF10 Q5 measured 10.8 ns a row on a hash table and 2.3 ns on the bits, over 30M rows. The index is built in parallel, one part for each range of words. A build whose keys ascend in its row order (a scan of a table in primary-key order) needs no parts: each chunk of its batches sets the bits of its own key range, the build rows are the ranks, and there is no order (sortedRank); equal keys keep the start of each run in firsts. Other joins keep a map in 64 partitions, and above 262,144 keys a blocked Bloom filter (16 bits a key, two bits in one word a key) tested before it: in a selective join most probe rows miss (StarRocks' runtime filter).
• The inner and semi joins on one integer column of a scan cut its morsels first (joinFirst): each tests that column of every visible row against its build (rank bits, else range, Bloom filter and map), the scan's filter then reads its columns for the rows left, and only the rows that pass are read whole. A test that keeps most of a morsel's first 1,024 rows is left to its probe.
• A join that a plan computes in several places is computed once before the plan runs, with the columns its places read, and each place reads its rows (sharedJoins, a WORK leaf). The membership tests of TPC-H Q21 are reduced by the rows of the join they test, which was computed three times. pg_stat_gizdb.shared_joins counts these joins.
• t ASOF [LEFT] JOIN q ON t.k = q.k AND t.ts >= q.ts matches each row with the nearest row of its key at or before its time (or after, by the comparison). It uses binary search over the right side, grouped by key and sorted by time.
  • Integer clocks that agree and temporal units that agree use exact i64 buffers to order and search. Other types keep the SQL comparison.
  • Stable ties select the last right input row for backward searches, and the first one for forward searches.
  • Serial and parallel policies share the same probe implementation.

Statistical aggregates

These aggregates can merge in pieces:

• The covar_pop, covar_samp and corr family of PostgreSQL, and the regr_* family, over one co-moment state (Welford updates, Chan merges).
• ols(y, x1, ...): least squares by centered incremental QR. It returns '{b0,b1,...}', or NULL when the inputs are collinear.

Rules:

• Correlation and R-squared normalize each centered moment separately. Thus a moment product does not overflow for finite large inputs.
• The shared finisher limits correlation to [-1, 1] and R-squared to [0, 1] when centered products differ a little through roundoff. This agrees with the mathematical result ranges and with the finisher policy of PostgreSQL.
• Native covariance-consistency checks still refuse larger violations before the finish.
• The intercept multiplies the finished slope.
• Sample variance and standard deviation use N-1, and give NULL at N < 2. Population forms use N, and return zero for one value.

Native moment state

The native MomentState API holds only the count, one or two means and the centered products. It uses typed nullable f64 rows, signed multiplicities that are not zero, and bounded lossless CRC checkpoints.

• Unit transitions keep their arithmetic order.
• Repeated insertions merge one constant partition. Repeated retractions apply one scaled correction of the centered matrix. Matrix work and state size do not increase with the number of copies.
• Population and underflow checks run before signed negation, also for i64 MIN.
• A complete retraction of a repeated row also requires the matching centroid and zero centered products. This is not a proof of old-image membership.
• Insertions and partition merges share the co-moment owner. Native partition merges normalize population ratios before centered products, to prevent a false intermediate overflow or underflow.
• This explicit native policy has its own history of partition rounding. Ordinary SQL merges keep their arithmetic order.
• Result definitions use the variance, stddev, covariance and regression finishers.
• Retractions require old-image attestation from the only membership owner of the caller. Their floating rounding history is different from a cold ordered SQL replay. Thus maintained SQL does not select this inverse silently.
• Finite-range, covariance-consistency and cancellation checks can refuse removals that are mathematically valid, and require an explicit rebuild. They do not guarantee arbitrary accuracy, and they do not infer membership from sufficient moments.
• Empty and singleton centered products are zero. NULL rows are skipped. Inputs and results that are not finite are explicit native errors.
• The population has a limit of 2^53-1. Byte credit limits the fixed encoded state, not the costs of the allocator or RSS.

Aggregate continuation codecs

Ordinary finite aggregate kernels expose one bounded CRC continuation codec. It is bound to the trusted aggregate name, the star flag and the arity.

• COUNT, numeric prefix, decimal and Welford, BOOL, BITS, bounded representative Values, co-moments, QR and bounded explicit histograms write only their active fields.
• DISTINCT maps and collecting aggregates require external strategies.
• Sequential spill and reload keep the same fold, raw floating bits and integer overflow history. They do not make reordered floating partition merges equivalent.
• The caller owns temporary disk ranges and shared resource leases.
• aggregateKernelSize is the exact wire credit. aggregateKernelMatrixBytes preflights the matrix cells before an ordinary step that allocates.
• Neither counts allocator overhead, spare Vec capacity, simultaneous input and output buffers, or process RSS.
• Invalid definitions, shapes, checksums and byte allowances fail explicitly.

The finite grouped cursor stores COUNT, SUM, AVG and TOTAL kernels that the registry admits separately from generic Acc kernels. Small references to plan positions connect them.

• COUNT has no numeric payload. Numeric kernels keep one canonical scalar payload. They do not keep unused variance, representative, model or histogram fields.
• The step, finish and continuation owner is used through a transient Acc bridge. There is no alternative integer-prefix, decimal or floating arithmetic.
• Interleaved generic variance, MIN and BITS, and all output positions, keep their semantics.
• The native running-window SumCtx of SQLite stays on its route.
• Shell and overlap credits do not change while that bridge is necessary. No smaller admission limits and no gains in physical RSS or latency are claimed.

Histogram continuation

Histogram continuation stores native boundaries, exact bucket populations, an optional raw floating sum, the interval and temporality, and the source provenance directly. It has no JSON shadow and no reconstructed observations.

• Geometry and provenance widths are checked before vector allocation. The native limits stay 4096 boundaries, 4096 sources and 65536 identity bytes.
• Canonical histogram validation owns the interval, bucket, duplicate-source and sum contracts.
• Shared resource admission must also cover decoded vector capacity, provenance validation maps and hash scratch, and simultaneous wire, input and output buffers. The wire allowance alone is not that resource limit.
• Reload keeps the sequential histogram merge history. It does not promise rounding of optional floating sums that is independent of order, or exact percentiles from buckets.
• Native source fixtures cover reload at each merge, raw sum bits, and malformed widths, UTF-8, checksums and definitions.

aggregateKernelScratchBytes gives a conservative transient credit for native histograms, from the wire allowance of the caller. It is capped by the same geometry and provenance limits before the arithmetic.

• It covers decoded geometry and text slots, growth of the validation map, sorted provenance and Value vectors, and the overlap of tuple and hash scratch. Its maximum is 6,561,824 logical bytes.
• It refuses smaller lease allowances of the caller, and kernel strategies that are not supported.
• The executor must get this shared resource lease before it decodes or validates. It must keep the lease until the decoded point and the temporary work are released, independently of the input and output wire leases.
• A fold that grows must reserve against its configured maximum state-wire allowance, not only against the length of the last decoded frame. Incoming histogram parsing and merge operands need their own overlapping leases.
• This credit is an explicit conservative admission policy. It is not a measurement of live allocation or RSS, and it does not permit the omission of other aggregate, input or output reservations.

Compact counted keys

Compact counted-key strategies make zero or one key a small payload. They allocate the Ordered index only for a second distinct key.

• A removal back to one key releases the promoted index in the new version. Immutable earlier versions keep their own state.
• COUNT DISTINCT, extrema, exact weighted percentile and MODE use the same canonical key, multiplicity, comparison, weighted summaries and codec and cell contracts. There is no second tree or checkpoint format.
• compactKeyReservation owns the conservative physical credit for one version of this representation. The executor adds its membership, receipt, transient and other state credits.
• Logical reservations and local work counters are not measured allocation, RSS or latency.

OLS

OLS folds centered rows with Givens rotations, and merges triangular factors without a store of observations.

• Its triangular solve equilibrates predictor units by exact powers of two.
• It keeps its successful finite compensated rows.
• Temporary normalized coefficients that are larger than the floating range use mantissa and exponent compensation in the same term order. Predictor units are restored before the coefficients are materialized.
• Very small independent RHS channels and offdiagonals are not erased by one global scale.
• This is compensation at finite precision. It is not arbitrary precision, and not a guarantee for every ill-conditioned design.
• Final coefficients that cannot be represented stay explicit errors.
• Native source fixtures pin wide units, cancellation of large terms and subnormal units.
• The numerical rank test keeps the squared limit of 1e-12.
• Complete input rows must be finite. NULL rows are skipped. A factor or coefficient larger than the finite f64 range is an explicit error.

One private compensated linear-combination owner serves triangular back-substitution, intercept reconstruction, inference loadings and covariance, and point prediction.

• It keeps small terms across cancellation, without a second model or a persistent cache.
• Outside the adaptive triangular solve, terms and partial sums that are not finite stay explicit errors.
• It cannot recover slope information that an ill-conditioned design already lost. It does not promise exact arithmetic past the precision of finite f64.
• The private triangular solver returns explicit range errors to fit and inference. Coefficient scaling does not change, and gives no finite substitute for an overflow.

Centered QR insertion and merge keep their finite-difference arithmetic. When the subtraction of finite opposite extremes alone overflows, the owner normalizes mean weights and scales the observations before the subtraction, for the centered factor. This admits means and factors that can be represented, without raw observations or a second model. Factors that really overflow still fail.

ols_summary

ols_summary(y, x1, ...) reads the same augmented factor. It returns JSON text with these fields:

• Covariance and standard errors use the classical unweighted homoscedastic least-squares estimator.
• Coefficients and standard errors use response units for the intercept, and response/predictor units for each slope. Each covariance entry has the product of its two coefficient units.
• At zero residual degrees of freedom, coefficients and RSS stay available. sigma, covariance and std_errors are JSON null.
• Both model results return SQL NULL for deficient rank or too few complete rows.
• An RSS or covariance that cannot be represented is an explicit summary error. The result with coefficients only does not require a finite RSS.

ols_predict

ols_predict(model, x1, ...) returns a FLOAT point prediction from the intercept-first coefficients of a summary.

• A NULL model or predictor returns NULL.
• The number of coefficients and the full rank must agree with the given predictors. Coefficients and numeric predictors must be finite.
• Compensated summation keeps small terms across cancellation. Products, partial sums or results that are not finite are explicit errors.
• Broadcast and dictionary text models are decoded lazily, one time for each distinct entry in the batch, after its first complete predictor row. Other model representations use scalar decoding for each complete row. No persistent model cache is kept.
• Prediction intervals, formula evaluation, weights, hypothesis tests, and compatibility with formulas, packages and the R package ecosystem are outside this contract.

Percentiles

• Percentiles validate their direct probability, also over an empty or all-NULL input. They return NULL for a NULL probability. They interpolate opposite finite extremes without an overflow of their difference.
• Literal, parameter, cast, scalar-expression and uncorrelated scalar-subquery probabilities are evaluated one time for each group.
• Direct expressions can read grouping keys. Other input columns and nested aggregates are refused, also in dialects that permit ordinary bare columns.
• Ordered percentiles and mode keep ASC and DESC, also for discrete ranks and mode ties.

Bounded external aggregate finalization

Ranks

finishAggregateRanks uses the percentile probability and interpolation owner. It seeks a maximum of two non-NULL ranks that the external sorted-input operator gives.

• That operator certifies, before it seeks, the permitted argument families of the full input, and the identical comparison, direction and representative order. Reads of selected ranks cannot validate input that was not seen.
• The native rank population has a limit of 2^53-1. It uses inherited floating probability arithmetic, not exact rational rank semantics.

Finite cursors:

• A cursor can call prepareAggregateRanks one time, and use aggregateRankRequests to capture zero, one or two selected values across bounded input polls.
• The returned position has the same descending direction and floating interpolation meaning. The cursor owns the certification of the complete non-NULL population and the comparison order.
• finishAggregateRankSelection requires values in request order, and uses the only percentile interpolation owner.
• The callback finisher delegates to this API. It still refuses an invalid first value before it reads a second value.
• No sample vector of the population size and no repeated rank scan is necessary.

diskRankGroupedFinite

The explicit diskRankGroupedFinite strategy uses the external SORT and the shared Resources owner, for one median or percentile aggregate without DISTINCT over a proven stored INT column.

1. It counts and type-certifies each complete group.
2. It visits the sorted run of that group again, to capture the canonical one or two requested ranks.

• Both passes yield after the configured source-row work credit, also for one hot group.
• It keeps only bounded start, end and current reader pages, group keys, and a maximum of two selected Values.
• Direct probabilities first admit group columns, constants and parameters. Computed inputs and types that are not proved refuse admission.
• NULL, filter and direction rules, and interpolation, stay in their owners.
• The SORT preparation is synchronous. Bounded rank polls do not give bounded constructor latency or a scan route for persisted tables.

Distinct keys

finishAggregateDistinctKeys counts runs of distinct canonical equality keys through a monotone AggregateKeyPull callback (Result<Option<bytes>, string>).

• Its only capability owner admits COUNT and count_big, and the exact intrinsic definition of approx_count_distinct.
• The external run owner makes Table keys, removes SQL NULL, sorts bytewise, certifies the population, and owns IO and input leases and the enforcement of work and deadlines.
• Finite executors can use beginAggregateDistinctKeys, stepAggregateDistinctKey and finishAggregateDistinctKeyFold to pause after any input page.
• KeyFold keeps only counters, declared limits and the previous opaque key. Refusals occur before mutation. Immutable earlier poll versions stay valid.
• The pull wrapper delegates to the same fold. One transient canonical COUNT accumulator is made only at finalization.
• No value vector or deduplication Map grows with the population.
• It checks the sort order, the declared key size, the exact end of input, callback errors and the result-wire allowance.
• aggregateDistinctKeyCredit reserves a conservative logical overlap of 2,048 + 8 * maxKeyBytes before the pull, which includes both key buffers.
• SUM and AVG DISTINCT do not use this unordered path. They require the restoration of the first retained ordinal, to keep the prefix and floating history.

Pulled values, MODE and collections

finishAggregatePull uses a monotone IO callback (Result<Option<Value>, string>) for MODE, string concatenation, and array and JSON-array collection.

• It checks the declared population and the final end of input, and keeps IO errors.
• It validates the MODE order, and keeps the first-mode tie rule and the typed representative.
• The caller gives the first retained separator or the MODE direction, and applies the canonical NULL, filter and DISTINCT rules before the pull.
• finishAggregatePairs separately uses a monotone callback of key and value pairs, for the JSON object aliases.

MODE also exposes beginAggregateMode, stepAggregateMode and finishAggregateMode for finite polls.

• One fold keeps its best and previous typed values, and checked run and row counters. It keeps the first sorted mode when populations are equal.
• NULL, input limits, decreasing order and exhausted counters are refused before mutation. Immutable poll versions stay valid.
• Ordinary MODE and the pull wrapper share this kernel and finisher.
• The MODE fold also has a bounded continuation codec. It binds a new trusted direction and allowances, and keeps the exact typed best and previous values. It uses the canonical admitted Value decoder. No input replay or population array is encoded.

String, array and JSON collection exposes beginAggregateCollection, stepAggregateCollection, stepAggregateCollectionPair and finishAggregateCollection, through the same canonical writer.

• A cursor can pause between scalar or key and value inputs. It does not have to use a full hot group during finalization.
• Each step renders under the scalar scratch credit. Then it checks the complete appended entry and the necessary closing delimiter, before it changes the output or the population.
• Refused inputs keep the earlier continuation.
• Duplicate JSON keys, NULL values, separators and dialect rendering keep their meaning.
• The configured output wire limit stays mandatory. The resident mode still requires an equivalently large retained buffer.
• aggregateCollectionOutputCredit credits overlapping output carriers and separator storage. Scalar rendering, source pages and other operators need their own grants in the same Resources scope.
• This source implementation adds native cases for pages, COW, refusal and closed output.

Collection chunks

beginAggregateCollectionChunks uses the same writer and renderer with one bounded pending entry.

• nextAggregateCollectionChunk gives raw byte fragments.
• finishAggregateCollectionChunks queues the closing framing.
• collectionChunkCertificate certifies the complete result only after it drains.
• Continuation codecs keep pending fragments and the trusted definition and context. They never keep output that was already emitted.
• Chunks can divide UTF-8. A receiver must keep them as bytes, and atomically publish the staged output with its final certificate.
• Individual scalar limits, and separately admitted render, source and page buffers, stay mandatory.

The native diskCollectionGroupedFinite uses the external sort by equality group and source ordinal, and stages these fragments in TempPages.

• Polls charge input and chunk work. Only a complete group whose framing certificate and temporary arena seal both succeed exposes its typed keys and certificate.
• readDiskCollectionPayload returns byte pages leased separately. closeDiskCollectionDelivery requires that the certified payload is drained before the next group starts.
• This cursor keeps no complete output buffer.
• It admits one scalar or pair collector in source order, bounded column and constant arguments and filters, and constant or parameter separators. DISTINCT and explicit aggregate order stay checked refusals.
• The caller owns each returned page until close.
• Source, render and chunk overlap use the same Resources issuer. A temporary quota failure leaves the output not certified, and requires cursor cleanup.
• The logical plan that the caller owns can still keep input VALUES. Sort construction is synchronous.
• This native delivery path does not give a Batch or PG scalar reader, a durable cursor restart, or atomic publication of a maintained view.
• The u32 text payload and the configured output wire caps stay in force.

Selection, pair and render credits

aggregateSelectionCredit(maxValueWireBytes, leaseAllowance) preflights the conservative logical overlap 4,096 + 12 * maxValueWireBytes. This is for two retained scalar values and the incoming scalar, each with four times the buffer capacity.

• This covers rank selection and MODE. The cursor owns shared leases, the cadence of finite work and deadlines, and output consumption.
• New independent native cases include 32 bounded pages, limits of NULL, probability and precision, descending ranks, coalesced endpoints, binary discrete and MODE results, overflow and COW, and refusal boundaries.

The JSON object pair aliases keep the conversion of key strings, value rendering and the input order of duplicate keys.

• No object map and no key or value arrays of the population size are made.
• These aliases do not give the duplicate-key normalization or NULL-key compatibility of PostgreSQL JSONB.

Resident collection finalizers share the transient object and array writers and the canonical SQL and JSON renderers. There is no second aggregate state or sort.

• Result limits count the exact canonical Value wire bytes, and refuse an overflow without truncation.
• aggregateRenderCredit preflights a conservative credit for each scalar before rendering. This includes worst-case JSON escapes, vector JSON shells and overlapping render buffers.
• Input, separator, output and decode buffers, container capacity and the final output conversion need concurrent leases in the shared Resources owner of the caller.
• MODE can keep two scalar references. Percentile interpolation can keep two rank values.
• Credit is not measured allocation or RSS.
• The result still occupies a bounded resident Value. A stream of a result to a client, or a spill of its output, is a separate physical sink contract.

Compact maintained aggregates

The function owner separates transitions that the executor attests from its one arrangement of full-row membership.

• Ordinary COUNT keeps only counters, and finishes without an enumeration of rows.
• COUNT DISTINCT keeps canonical equality keys with counted multiplicities in the Ordered index. A delete removes a key only when its last copy goes. It is exact, not a sketch.
• The historical spelling approx_count_distinct uses the same exact compact strategy.
  • Its aggregate name stays part of the checkpoint definition.
  • Intrinsic distinct normalization applies also without an explicit SQL DISTINCT modifier.
  • NULL is excluded. An empty group returns integer zero. The last counted copy removes a canonical equality key.
  • It is not an approximate algorithm. It gives no sketch error limit and no measured capacity claim.

State records

• Native compact APIs accept and return CompactAcc (count, state, inputError), not the ordinary Acc record.
• Retained signed APIs use SignedAcc (state, inputError), with the counted membership and the bounded replay contract.
• Neither retained strategy stores unrelated ordinary numerical or model fields. Ordinary Acc no longer contains either state.
• Signed callers read .state in place of .counted.unwrap(). Compact callers read .state in place of .compact.unwrap(). This is a change of the native source API. Checkpoint versions and their encoded meanings stay the same.
• Signed reorder and merge refuse earlier errors before mutation. This storage separation does not make replay local to a delta.
• Optional counted and numeric strategy records are in one immutable Vec cell only when admitted. Ordinary COUNT holds neither strategy payload, and reserves no inline Ordered or AVL record.
• Strategy changes keep earlier versions immutable through the same Vec ownership rules.
• This layout specialization alone does not give allocator byte costs or an RSS improvement.

Replay

A transient AggregateReplay fold can use the single attested group membership of the executor. Then each fallback aggregate does not keep argument tuples and identity and ordinal indexes.

• beginAggregateReplay, stepAggregateReplay and finishAggregateReplay use the ordinary accumulator and finisher. The standalone finishSigned uses the same fold.
• Copies stay bounded replay work. Ordered members must arrive in strictly increasing declared unique order.
• DISTINCT keys and typed-tie checks stay transient fold state.
• Shared membership admission excludes the MIN/MAX typed-representative fallback, collected aggregates, and histogram merges with an order-sensitive optional floating sum. It requires deterministic arguments and filters from the executor.
• Logical input and kernel-row counters and charged wire work are not allocator, RSS or latency measurements.

SUM and AVG

SUM and AVG keep only a numeric scalar, the multiplicity, the identity, and a stable binary SQL order key for each contribution. They use an AVL tree of exact totals and relative prefix envelopes.

• Updates and key moves change a logarithmic number of summaries, in fixed pages of 64 nodes. Deleted slots are used again.
• Staged values keep earlier snapshots, because changed node pages are copied.
• Numeric pages and the ordered identity index share the persistent 32-way PageDirectory of the SDK. Thus a staged page change copies bounded directory vectors along one radix path, and not every unrelated page pointer.
• Ordered node slots and free IDs use the same owner. Pop releases empty directory tails, and keeps immutable earlier versions.
• This source structure removes the flat directory copy.

Finish:

• Safe integer and decimal results finish from the root summary. Integer prefix overflow follows the batch aggregate contract.
• A decimal summary finish requires a proof that no canonical 40-digit operation loses scale. If not, the ordered scalar fold is used.
• The FLOAT, decimal rounding and integer overflow fallbacks charge their count of surviving scalar replays explicitly through compactFinishWork.
• There is no reassociation, and no claim of floating retraction in constant time.
• DISTINCT SUM and AVG, and arguments that are not numeric, are outside this compact strategy. They require an explicit alternative admission.
• The executor gives unique binary order keys that agree with the comparator. The order of a physical scan or commit is not a substitute.

Persistence:

• Header and per-key cell APIs let one transaction persist only the changed compact cells, with state, output and cursor metadata.
• Recovery streams bounded cells and checks totals. The full encodeCompact is a snapshot operation.
• Byte budgets give the canonical serialized payload, not RSS. Optional tree state leaves ordinary batch accumulators without allocations for each row.
• Transition and summary-update counters give executed work, not accepted benchmark speedups.

The native finisher is finishCompact(acc, cx, direct). Callers give the explicitly evaluated direct arguments of the group output, or [] when there are none. There is one result owner. No overload, default parameter or alternative percentile finisher hides these inputs.

compactPageWork reports the unique touched node pages, the newly made pages, the exact added retained wire bytes, the logical directory nodes written, and the earlier slots in their bounded directory vectors. Directory work includes the ordered identity index. Wire growth is not an allocator or RSS counter.

Compact percentiles

Compact median, percentile_cont and percentile_disc over enforced INT values keep one counted canonical key for each distinct value in the Ordered B+tree.

• Optional summaries of weights that are not negative support rank selection weighted by multiplicity in logarithmic time, in the same page pool. Ordinary maps keep empty summary vectors, and keep their signed payload meanings.
• Insert, replacement, delete and restore update population summaries through the same tree owner.
• Checks of population overflow and negative weights occur before mutation. Earlier versions stay immutable.
• This stores exact retained counts. It is not an approximate sketch, and not reconstructed samples.
• The ordinary and compact finishers share probability validation, NULL and empty handling, DESC direction and interpolation, which includes the finite opposite-sign rule. Their probability multiplication stays rounded floating arithmetic, not exact rational arithmetic.
• compactPopulationLimit owns the admitted configured limit of 2^53-1 for this strategy. Larger configured populations, and DISTINCT, dynamic or other-family inputs, keep signed admission or get an explicit refusal. Weighted Ordered itself checks exact i64 populations up to i64 MAX.
• GIZDB-COMPACT-RANK-1 and the scalar cell wire owner persist counted keys. Recovery builds the weight summaries again while it streams bounded cells.
• Finish reads a maximum of two weighted ranks, and expands no duplicate arguments.
• Optional descriptor and node storage and logical path counters are not proof of allocation, RSS or latency.

Compact MODE

Compact MODE over enforced stable INT values uses the same counted scalar keys and weighted Ordered tree as compact percentiles.

• Each weighted child summary also records its largest multiplicity. atPeak selects the first or last key at that multiplicity, without a scan of keys or an expansion of duplicate samples.
• This keeps the smallest-value tie rule of the ordinary MODE finisher, and the largest-value tie rule under DESC.
• NULL and empty behavior do not change. Counts stay exact up to i64 MAX. No floating rank calculation is used.
• GIZDB-COMPACT-MODE-1 and the scalar cells persist one key and count for each distinct value. Streamed recovery builds both summaries again.
• Proof of the stable type is the responsibility of the query owner. DISTINCT, mixed or unproved input families keep the signed strategy.
• Weighted inner summaries now contain two i64 fields. Ordinary maps still allocate none.

Ordered reservation

Ordered owns a conservative reservation(includeLeaf) for one version. Consumers do not duplicate guesses of tree fanout or node size.

• Touched node writes keep separate credit for leaf and separator key buffers. Reservation reads do not scan the index.
• includeLeaf=false omits only the leaf payload that the canonical bag owner of a caller already charged. All separator references stay charged.
• Structural credit includes bounded node vectors, optional population and peak slots, pool buffers and generic directory paths.
• Checked saturating arithmetic returns i64 MAX on impossible accounting. This prevents an admission after wraparound.
• This is a conservative logical resource reservation. It is not measured allocator or RSS bytes, allocator overhead, or totals for historical versions that are kept at the same time.

Bit aggregates

Ordinary bit_and and bit_or keep exact AND and OR masks in the scalar fields of the accumulator. They share one step kind and mask merges in constant work.

• Each argument is converted one time through the canonical integer conversion. No collecting vector is stored.
• Its NULL result finisher also serves compact bit populations.
• Signed replay folds each retained argument one time, whatever its duplicate multiplicity is. It keeps the non-NULL count.
• Sliding bit windows can use the merge tree after the query owner admits this scalar kernel. Until that predicate is integrated, they still fold each frame explicitly.

Native compact bit_and and bit_or keep 64 exact one-bit population counters in an optional bounded payload. Other strategies do not have it.

• It keeps no argument values and no value index.
• Each transition updates a maximum of 64 populations. The finish reads those counters, and does not replay rows.
• Signed conversion uses the numeric owner, which includes the sign bit. Empty and all-NULL groups return NULL.
• DISTINCT does not change a bitwise result. The single membership arrangement of the executor validates duplicate and old-image multiplicities.
• Only enforced stable INT sources admit this SQL strategy. Other inputs keep signed replay.
• GIZDB-COMPACT-BITS-1 records the rows, the non-NULL count and all 64 populations in a bounded CRC header with zero cells. Wire accounting includes the full payload before it is allocated.
• A zero compactFinishWork means no replay of surviving rows. It does not mean zero counter work, and it is not an allocator, RSS or latency measurement.

Boolean aggregates

Native compact bool_and, every and bool_or use exact counters of the true, non-NULL and total populations, without an index or a retained scalar payload.

• CompactAcc.count is the true population for this strategy. state.nonNull and state.rows keep their ordinary population meanings.
• The shared Boolean result finisher also serves batch and signed replay.
• Empty and all-NULL inputs give NULL. A delete of the last true, false or non-NULL copy keeps the SQL meaning.
• DISTINCT has the same Boolean result. The canonical source membership validates multiplicity one time in the executor.
• Func.compactStrategyForType admits this strategy only for enforced stable BOOLEAN input. Dynamic SQL inputs keep the signed strategy.
• Native transitions use the canonical truth owner. They refuse binary arguments, and validate true, false and NULL underflows before mutation.
• GIZDB-COMPACT-BOOL-1 records all three populations in one bounded CRC frame. The header API uses that frame with zero cells. No metadata guess discards non-NULL counts.
• Constant population arithmetic does not give an end-to-end latency claim or a product parity claim.

Compact MIN and MAX

Native compact MIN and MAX keep one counted scalar payload for each distinct raw value in the Ordered index, after the executor validates the source membership one time.

• A delete of the last copy removes that scalar. Finalization seeks the first or last surviving entry, and does not replay rows.
• NULL contributes only to population accounting. Empty and all-NULL results stay NULL.
• Homogeneous families are admitted: integer, float, decimal, binary text, bool, BLOB, and temporal values of the same unit.
• These are refused explicitly, and need the signed strategy with a suitable order contract: mixed coercive families, vectors, DATE microseconds that cannot be represented, and equal values with different raw typed representatives (decimal scale, signed zero or NaN payload). This keeps the first typed representative of cold MIN/MAX, and does not choose an arbitrary byte tie.
• A nonbinary collation requires separate admission evidence.

SQL eligibility:

• Func.compactStrategyForType is the SQL eligibility owner over the only compactStrategy declaration.
• SQL first uses compact extrema only for enforced stable INT, BOOL, binary TEXT and BLOB expression types. FLOAT, DECIMAL, temporal and dynamically typed input keep the signed strategy until preflight and fallback keep evaluated arguments and typed representatives.
• A native transition refusal does not permit the refusal of a SQL query that is now valid.
• The caller must prove the stable type contract before it selects the compact strategy.

Persistence:

• The header and cell streaming APIs carry raw byte keys. Extrema use cell kind 2 and full GIZDB-COMPACT-EXTREMA-1 snapshots.
• Canonical orderBytes framing and Value wire encoding own the typed key. Recovery computes it again, and verifies the family, multiplicities, totals, CRC and limits.
• A conservative key scratch limit and exact retained wire credit are different from allocator or RSS evidence.
• Directory counters give bounded path copy, not latency or physical memory.
• These contracts do not give parity with R or with database products.

Typed metric analytics

The statistical function owner exports MetricCounter and MetricHistogram, with validation, conversion between cumulative and delta, counter rates, spatial histogram merge, and nearest-rank quantile containment. SQL uses the same implementations through the metric functions.

Carriers

• The versioned JSON text carriers use giz.metric.counter.v1 and giz.metric.histogram.v1.
• Timestamps, counts and i64 counter values are canonical decimal strings. Thus values above 2^53 stay exact.
• The counter number_type is i64 or f64. Floating values must be finite.
• Decoding validates the carrier through the SDK pull reader. It keeps only the final value span of each declared field, so the last duplicate key wins.
• Overwritten and unknown nested values are syntax-checked without a JSON tree.
• Native array widths are checked before growth. Escaped strings get raw-span and decoded-byte checks before admission.
• The carrier has a limit of 1 MiB. The limits of 4,096 bounds, 4,097 counts, 4,096 sources and 65,536 total identity bytes stay.

Decode and overlap credits

metricDecodeScratchBytes(value, leaseAllowance) preflights a conservative logical credit of 65,536 + 64 * carrierBytes. This is for scanner and field metadata, input and slice, and the overlap of bounded decoded vectors and strings.

• It validates the carrier type and size, and the allowance, before the parse.
• It is not an allocator or RSS measurement.
• Disk execution must get this shared-scope grant before the decode. It must separately cover canonical native validation, the resident point, merge operands, and simultaneous encoded input and output.
• Disk histogram admission stays refused until those overlapping grants are attached. The decoder alone does not prove that path.

aggregateKernelOverlapBytes(name, star, arity, stateWireAllowance, inputCarrierBytes, outputWireAllowance, leaseAllowance) owns a combined histogram grant for the shared query lease of an operator.

• It covers pooled, incoming and merged native points at full native caps, typed carrier decode, validation, merge and renderer scratch, and simultaneous state and result wire buffers.
• Full caps are necessary because native growth and refused rendering can occur before the configured wire and output check. A very small output limit never makes that credit zero.
• Larger configured carrier and frame limits saturate at the native carrier cap before the arithmetic.
• The maximum credit is 107,848,052 logical bytes. The helper does not prove allocator, RSS or disk-path use.
• The operator must enforce its input carrier limit, stage only successful state, keep the grant through rendering and typed output, and release it through the resource lifecycle.
• The native parser, merge and finisher stay canonical.

aggregateKernelTransitionBytes(name, star, arity, acc, inputCarrierBytes, outputWireAllowance, leaseAllowance) also derives an adaptive overlap from the actual retained shape of buckets, sources and identities, and from the admitted raw input carrier limit.

• It checks retained structural sizes without an allocation of validation maps or hashes.
• Raw carrier bytes give a conservative limit for incoming decoded identities, array geometry, and also invalid empty-source entries. Native caps limit merged growth before any final size check.
• The same native, decoder, renderer and sealed-frame size owners charge the old, incoming and merged overlap.
• Refused rendering stays charged, also with an output allowance of one byte.
• The unchanged full-cap grant stays available as a conservative alternative.
• A new state with an input carrier of 300 bytes and a result allowance of 65,536 bytes reserves 701,848 logical bytes, not the global native maximum.
• This is an arithmetic admission contract. It is not an allocation, RSS or latency measurement.
• The executor must resize its shared lease before decode and step. It keeps unrelated source, sort and aggregate grants, and keeps the output overlap until consumption.
• A zero incoming carrier can preflight finalization. It does not discard the geometry of the retained point.

Counters

• series declares the identity of resource, scope, name and attributes. unit is part of the identity.
• start and time are nanoseconds since the Unix epoch, not negative, with a limit of signed i64.
• temporality is delta or cumulative. Counters also declare the boolean monotonic.
• Counter differences keep i64 arithmetic. They refuse overflow, decreases without a monotonic reset, changes of identity, unit or representation, and overlapping reset epochs.
• A cumulative point with an unknown start (start=0), or an initial reset with zero duration and an unknown start (start=time), gives no initial delta or rate. Keep that original point as the baseline of the next conversion.
• Resets with a known start give the total of the new epoch.
• Delta accumulation requires contiguous known intervals. Gaps, overlaps and missing starts are explicit errors.
• Counter rates are floating values for each second, in the unit of the counter.

Histograms

• bounds are finite numbers that strictly increase. counts has bounds+1 exact i64 entries, not negative, whose checked total is count.
• Upper bounds are inclusive. Lower bounds are exclusive. Empty bounds with one count are one unbounded bucket.
• sum is optional, finite and not negative. It is zero for an empty population. Omit it for signed observations. A missing sum stays missing during conversion or merge.
• Histogram conversion refuses changed geometry and decreases without a mark.
• A spatial merge requires identical intervals, units, geometry and temporality, and disjoint declared sources.
• Pooled results keep the union of the sources and a canonical population identity. Thus an attempt to merge them again with a constituent is found.
• Producer provenance is an assertion of the caller. It is not authentication.
• The aggregate stores one histogram in the accumulator, propagates validation errors, skips SQL NULL and returns NULL when there is no input.

Merge order:

• Bucket populations and the provenance union are exact.
• An optional f64 sum follows pairwise IEEE pooling, and can change with the order of arrival or with partition groups. A native vector merge sorts the given series, but pooled summaries cannot restore an original order of summation.
• Thus the canonical aggregate property marks a histogram merge as order sensitive. A continuous signed route requires its unique total-order admission contract, although bucket populations are exact.
• This does not permit the refused disk histogram route. Shared-membership replay keeps the retained route of this aggregate. No scalar sample values or extra sum state are reconstructed to hide that numerical difference.

Quantiles:

• Quantiles return JSON bucket containment: lower, upper, inclusivity, and exact=false.
• They use an exact rank calculation with binary mantissas and integers, also above 2^53.
• A null endpoint means an unbounded tail. Empty populations return SQL NULL.
• Bucket counts cannot identify an exact sample percentile. Thus no interpolated sample or finite tail error is invented.

Limits and scope:

• Admission caps are 4096 boundaries, 4096 population sources, 64 KiB of aggregate identity bytes and a JSON carrier of 1 MiB. These are limits of encoded input. They are not an RSS guarantee.
• Optional min and max, exemplars, flags, exponential histograms and OTLP transport are not part of this carrier. JSON fields that are not supported fail explicitly.
• No parity with Datadog or with complete OTel is claimed.
• Interval and bucket semantics follow the OpenTelemetry metric data model and the OTLP metric schema.

Time buckets

time_bucket(interval, ts [, origin]) rounds a time down to buckets from 2000-01-03 (months from 2000-01-01), as TimescaleDB does.

Column kernels

Column work runs in SIMD kernels (db/core/kernels.giz, 8 lanes of 64 bits):

• comparisons of a column with a constant or with a column;
• + - *, with the overflow check of the scalar path;
• AND and OR in three-valued logic, and IS NULL;
• selection vectors;
• count, sum, min and max for one group (NaN is the greatest, as PostgreSQL orders it);
• decimal columns of one scale against a constant, and decimal + - * on their coefficients (exact. Past 64 bits, the row path rounds as NUMERIC does);
• EXTRACT one time for each distinct date;
• vector distances to a constant vector;
• segment packing (bench/db olap).

The machine decides, never a constant written for one host (gizlang:machine, docs/internals/compute.md):

• A compiled kernel unit is built for the x86-64 level of the host (-march=x86-64-v4 where AVX-512 is usable), and a process skips a cached unit above its level (GIZLANG_CPU caps both).
• Cache-sized tables take their size from the core's share of the last-level cache.
• pg_stat_gizdb_machine shows what GizDB sees, one name, value row a fact: arch, level, features, cpus, threads, cache_groups, nodes and lN_share (bytes of cache level N a core may count on).

Source batch 2026-10-11:

• Integer SIMD sums prove from the sign bounds of each value that every ordered prefix fits. Values outside that proof use the ordered row fold. Thus lane cancellation cannot hide an integer overflow.
• Integer aggregate states keep relative prefix minimums and maximums and an exact total. Only pieces that overflow allocate wide decimals.
• Ordered piece merges translate the full prefix envelope. Thus an intermediate overflow cannot disappear through cancellation, and does not occur only because a later piece is aggregated alone.

Performance evidence uses bench/db/roofline/run.giz and bench/db/tpch/run.giz.

• They check complete nullable result cells, exact reference integer, decimal and text types, and finite floating tolerances.
• They record the first execution, and the median and range of five later executions.
• The Roofline denominator is a logical model of a full scan of the named columns. It is not an unconditional physical lower limit. Pruning, indexes, cache residency, intermediate materialization and actual parallelism need separate plan and buffer evidence.
• Historical measurements and assumptions stay in bench/db/README.md.

Session

session.giz runs autocommit, or BEGIN, COMMIT, ROLLBACK and savepoints, over the transactions of the engine, at each isolation level (§3).

• Every lookup, index probe and scan is recorded as a read for the commit to validate.
• A btree index answers equality on each prefix of its columns. A read through a prefix conflicts only with writes of that prefix. An index on (chat, seq) serves chat = $1.
• Statements: INSERT (defaults, identities, ON CONFLICT, OR REPLACE and OR IGNORE, RETURNING), UPDATE and DELETE (with FROM), CREATE TABLE [AS], indexes, views, sequences, ALTER TABLE ADD and RENAME COLUMN, and DROP.
• nextval('sequence') and currval('sequence') with a literal name use the same sequence owner as sequence.NEXTVAL and CURRVAL. Supported calls in the SELECT list and in VALUES share row numbers.

Numbers

db/decimal.giz and types.giz: DECIMAL, NUMERIC, NUMBER(p,s) and MONEY are exact decimals. They have a 128-bit coefficient and a scale, and use one i64 while the value fits. Thus 0.1 + 0.2 = 0.3, and a sum of cents is cents.

• A column keeps its declared scale (rounded half away from zero) and precision (22003 past it). A column without a scale keeps the scale of each value.
• Addition and multiplication are exact. Division takes the result scale of PostgreSQL. Past 38 digits, a result gives up fraction digits, and then fails (22003).
• Columns whose coefficients fit 64 bits use the batch kernels. Wider columns use the exact row path.
• Integers, floats and decimals order exactly against each other (one key encoding).
• As in PostgreSQL, a decimal with a float compares and computes as a float. CASE and COALESCE arms of mixed number types are cast to the type that they resolve to.

• Floats read NaN, Infinity and -Infinity.
• SQLite keeps its affinity (NUMERIC holds reals).
• PostgreSQL text-to-integer casts require a signed decimal integer spelling. They refuse fractional, exponent and junk text. Numeric-to-integer casts keep their rounding rule.

Binary values

JSON stays checked text. Native BLOB and BYTEA use Value.blob(bytes) and a packed column arena with row offsets and a separate NULL bitmap.

• Empty bytes are different from NULL. Equality and indexes keep raw bytes and unsigned byte order.
• The only tuple-key encoder returns opaque bytes. Native indexes, transaction write tracking, SQL joins, groups, sets and subqueries, and DISTINCT state use it.
• Identifiers stay checked UTF-8 text. Arbitrary keys never go through a string conversion.
• The WAL, schema and segment readers keep their previous tags. Binary adds Value tag 11, Type tag 9 and Column tag 9.
• PostgreSQL OID 17 supports exact binary parameters and results, and hexadecimal or legacy escaped text input. Text output uses \x plus lowercase hex.
• Numeric and implicit text operators refuse binary inputs. Explicit text casts render the canonical representation.
• Parquet byte arrays without an annotation keep bytes. STRING, ENUM and JSON annotations require UTF-8.

Native binary function admission

• BLOB and bytea stay raw through COUNT, exact DISTINCT, min and max, arbitrary-value and discrete order-statistic accumulation.
• Byte length and octet_length, explicit hexadecimal display, SQLite quote and typeof, PostgreSQL pg_typeof and sha256(bytea) operate on the original bytes, also with NUL and invalid UTF-8.
• Binary SHA256 returns binary digest bytes. Inherited text hashing still returns hexadecimal text.
• Explicit PostgreSQL to_json(bytea) renders the canonical bytea text in a JSON string. The parse of a model or JSON document from arbitrary binary bytes is not supported.
• Numeric, implicit text and unsupported collected-value functions refuse binary arguments.
• Ordinary and inverse-window numeric accumulators propagate input errors, also from a partition with a count of zero.
• Binary string aggregation, arrays and JSON collections, and continuous percentiles stay explicitly not supported.
• References for byte operations: PostgreSQL binary strings and SQLite core functions.

Kafka binary rows

Kafka payload and header rows use topic metadata version 2 and native binary columns. Text rows of version 1 require the explicit trusted Store.migrateLegacyIn(engine, tx, tenant, topic, maxRows, maxBytes, backing).

• It requires a clean caller transaction, a complete contiguous offset cut, the exact schema, the canonical legacy encoding, and positive independent credits of physical rows and bytes.
• Historical delta and segment versions are charged before scratch for the visibility scan is allocated.
• It stages no mutation when preflight refuses. It fences concurrent writes at every isolation level. It refuses other open snapshots at staging and commit, through the common quiescent-rebuild fence of the Engine.
• Converted rows and the unchanged next offset publish with the replacement schema and one Catalog revision, in the transaction of the caller. The caller commits and syncs through the durability owner.
• Larger cuts outside the selected credits are refused. There is no automatic conversion, partial publication or silent fallback.

Dialects

In dialect.giz, the dialect of a session is a value of the rules where PostgreSQL, Oracle, SQL Server, SQLite and MySQL differ in meaning. Examples:

• the empty string of Oracle as NULL, and its exact integer division;
• the type affinity of SQLite;
• LIKE case, NULL order and boolean text;
• DATE with time;
• what an error inside BEGIN does;
• the default isolation.

Every module reads the rule. Thus a dialect is a new value, not edits across the engine.

• A client selects it with SET sql_dialect, or options=-c sql_dialect=oracle at startup.
• Oracle clients connect over TNS on the same port as PostgreSQL clients (db/ora/server.giz): the first bytes tell the protocol. The server does O5LOGON against the role's Oracle verifier (set with its password) or the server password, then relays each statement into an oracle session. A session prepares each statement text once (up to 256), and every cursor of that text runs it with one Bind and Execute; a forwarding loop then answers a read or write by key as it does a driver's named statement. It holds a query's rows and serves fetches from the hold. A column described as text is fetched as text, and every other column in binary, so a value is never read as another kind than the one it is written as. tests/db/ora-server.test.giz logs on with the Gizlang client (db/ora/client.giz).
  • The server speaks the 12c and later TTC. JDBC thin 23, python-oracledb thin, node-oracledb thin and go-ora (field version 11) log on and run the same checks, and the server derives each client's wire form (token numbers, chunked strings, the end-of-call and ECID fields of a summary) from the field version the client announces. The logon answer carries the limits a client sizes its caches from (AUTH_MAX_OPEN_CURSORS), and NLS_SESSION_PARAMETERS, NLS_DATABASE_PARAMETERS and V$NLS_PARAMETERS answer the NLS queries drivers run at start. A statement can bind IN, OUT and IN OUT values, RETURNING ... INTO, and a REF CURSOR out; the DBMS_OUTPUT buffer is read with GET_LINE; DatabaseMetaData and DESCRIBE (function 119) are answered from the dictionary views.
  • A CLOB or BLOB column is described as a LOB (a LOB_MOD type modifier marks it between the SQL layer and the wire) and its value is a locator of the session. The server holds the LOBs a session fetched or created in memory (64 MiB, oldest data evicted first) and serves the LOB operations (length, read, write, trim, temporary LOBs, chunk size) on them. A select with LOB columns is described first, and the client then re-executes it with defines and a LOB prefetch, as Oracle does. A write through a locator of a stored LOB is not saved to the table; assign the value with an UPDATE.
  • The text of the ORA-nnnnn errors follows Oracle: ORA-00942 names "SCHEMA"."TABLE", an unknown function is ORA-00904.
• The text formats of the PG protocol stay those of PostgreSQL.
• One grammar accepts the spelling of every dialect (parse.giz).
• A PostgreSQL session also takes the MySQL and StarRocks functions and operators that analytics queries use, with StarRocks' answers and a type a driver reads as a number or a time: DATE_FORMAT and STR_TO_DATE with % elements, DATE_ADD, DATE_SUB, ADDDATE, SUBDATE and INTERVAL n UNIT (a date or a timestamp without a zone gives a timestamp), WEEKDAY, UNIX_TIMESTAMP (a wall time is the session zone's), UTC_TIMESTAMP, DIV, MOD, PMOD, CRC32, one-argument ISNULL, SUBSTRING_INDEX, FIELD, GET_JSON_STRING, GET_JSON_INT, GET_JSON_DOUBLE, PARSE_JSON, and the aggregates MAX_BY, MIN_BY (a NULL key is skipped), COUNT_IF and PERCENTILE_APPROX (exact, as percentile_cont). tests/db/starrocks-funcs.test.giz holds StarRocks 4.1's answers. CAST(x AS CHAR) stays PostgreSQL's char(1); write AS STRING or AS TEXT.

Routines

routine.giz, proc.giz, plexec.giz and trigger.giz give CREATE FUNCTION, PROCEDURE and TRIGGER, CALL, DO and anonymous blocks, in these forms:

• PostgreSQL (SQL, PL/pgSQL);
• Oracle (PL/SQL, :NEW and :OLD, DBMS_OUTPUT, RAISE_APPLICATION_ERROR);
• SQLite.

Execution:

• A function in a query is not interpreted for each row. A SQL function, or a PL function of declarations, assignments, IF, CASE, SELECT INTO and RETURN, becomes part of the query as an expression or a subquery (Froid, VLDB 2018). Thus the optimizer and the column kernels see through it.
• Any other function runs one time for each distinct argument tuple of the rows that the FROM and WHERE of the statement give. The plan reads its results by argument.
• Procedures, triggers and those functions run in the interpreter of the session. Each embedded statement is parsed and planned one time for each connection.
• Exception blocks undo their writes in PL/pgSQL, and keep them in PL/SQL, as each engine does. A failed statement undoes only itself.
• Triggers: BEFORE and AFTER, ROW and STATEMENT, UPDATE OF, WHEN, and a BEFORE trigger that changes or skips the row.
• PL/SQL follows Oracle Database 23 (the answers in tests/db/oracle/*.out come from it, known.txt lists the cases that differ): packages (a spec and a body, state kept for the session, an initialization block, overloads, calls without parentheses), collections (nested tables, VARRAYs, associative arrays, records, %ROWTYPE, their constructors and methods, BULK COLLECT, FORALL), PRAGMA EXCEPTION_INIT, PRAGMA AUTONOMOUS_TRANSACTION (the routine runs in a transaction of its own, which its COMMIT ends), GOTO and labels, and ORA-04091 for a row trigger that reads or changes its own table. A collection is a package variable, a parameter and a result (it crosses a call as one packed text, db/sql/types/collpack.giz); CREATE TYPE t AS TABLE OF e and VARRAY(n) OF e make a type of the schema; TABLE(c) of a variable or a function result is a FROM item with the column COLUMN_VALUE; x MEMBER OF c, subprograms nested in a block, RETURNING INTO, EXECUTE IMMEDIATE ... INTO/USING and FOR UPDATE with WHERE CURRENT OF work. Errors show the ORA-nnnnn texts of Oracle. A call qualified with a package name (pkg.total(x)) is the package routine, never a built-in aggregate of the same name.
  • Object types: CREATE TYPE t AS OBJECT (attributes, MEMBER and STATIC functions and procedures) with CREATE TYPE BODY, UNDER for a subtype, the constructor t(a, b), columns of an object type, x.attr, col.attr.method(), TREAT, and DROP TYPE ... FORCE. In PL/SQL an object variable has its attributes and calls its members; a procedure member writes SELF back. USER_TYPES, USER_TYPE_ATTRS and USER_TYPE_METHODS list them. A member call runs the method of the declared type (no dynamic dispatch to a subtype's override).
  • PIPELINED functions with PIPE ROW (read with TABLE(f(x))), WITH FUNCTION in a query, INSERT ALL and INSERT FIRST, an update of a join view, and ORA-06531 for a method on an uninitialized collection.
  • Not supported: collections of records in TABLE(), collection OUT parameters, dynamic dispatch of an overridden member.

Not supported: calls for each row of functions that are not inlined inside subqueries, and INSTEAD OF.

PostgreSQL protocol

In db/pg/, one event loop owns the engine. Each connection has a session.

• With --threads N (by default one per CPU the process may use), N more loops share the port, each on its own thread. A read-only statement outside a transaction runs there, against a snapshot that the loop of the engine publishes, when that snapshot holds every commit that was already confirmed to a client. All other work goes to the loop of the engine (bench/db).
• A loop also mirrors a transaction that the engine's loop holds (db/server/forward.giz). It starts at a plain BEGIN, or at a unit that has BEGIN before its first statement, as pgjdbc and the Oracle bridge send them (beginUnit).
  • The loop runs each later statement on its snapshot and sends its writes and reads as one write set. The engine checks the reads against its own state and takes the set (pg_stat_gizdb.shipped_adopted), or runs the unit itself.
  • A unit that the engine runs (FOR UPDATE, a named Parse, a failed set) carries a mark (SHIP_MARK). The engine answers it with the writes that the unit added (shipped_acked), so the mirror continues.
  • A point read by the whole primary key, of one or more columns, answers on the loop when its snapshot is the newest and the transaction did not write that row (local_reads).
  • A point UPDATE, INSERT or DELETE of a named statement by its whole key (integer, text or numeric columns) is a put or a delete that the loop makes without the statement path (shipped_rows), unless the transaction already wrote that row. An Execute row limit does not stop it (pgjdbc's executeUpdate sends 1): a write with no RETURNING runs whole, as in PostgreSQL.
  • When the loop's snapshot is the newest, it answers a write at once and sends the set after it (SHIP_AHEAD, shipped_ahead). A unit with BEGIN before its first statement answers at once too (SHIP_BEGIN). While such a write is not acknowledged, the reads that the loop answers go to the engine too. The engine takes the set, or runs the statements again and compares their answers with what the client got. A different answer makes the transaction fail at COMMIT with 40001, as a serialization failure does in PostgreSQL.
  • An error, the end of the transaction or a missing acknowledgement ends the mirror.
• It supports the simple and the extended query protocol (Parse, Bind, Describe, Execute, Sync), text and binary results, and SQLSTATE errors. psql, pgbench and gizlang:postgres connect unchanged.
• Group commit. A commit applies immediately and is written to the log. The log sync runs in a task of its own while the loop takes more commits, and the next sync covers all of them. A commit shows to other sessions once it is durable, as in PostgreSQL: a snapshot is the durable clock, so a read never waits for a sync, and only the reply of a client's own commit waits. A RESP session whose own commit is not durable yet reads the newest commit, as its writes do, and that reply waits for the sync too: a pipeline's GET reads the SET before it. A standby gets only the synced log. Thus no client sees what a crash can lose. (Under KV appendfsync everysec or no, a commit shows at once and RESP replies leave after the log write. As in Redis, the commits of all RESP clients of a loop pass, up to the point no connection is ready, go to the log in one write, and then their replies go.)
• changes('table', after [, wait_ms [, 'column', value]]) returns the committed changes after a position: lsn, deleted, then the row. It gives SQLSTATE GZ001 when the position was trimmed. With a wait, an empty answer parks the connection until a commit writes the table or the wait ends.
• A native watch (Access.observe, which Data.follow opens) keeps only its selection: the rows of one table that a filter picks. Each read takes that table's changes after its position through changes() with the filter as a column, as the watch's role may read them. Thus a watch holds no result state. A position that the WAL no longer holds, or a watch that expired, answers lost, and the page reads again. An UPDATE that moves a row out of the filter is not reported: the journal gives the row's new image only (a deleted row is its before-image).
• COPY FROM STDIN and TO STDOUT (db/sql/copy.giz), in the text and CSV formats of PostgreSQL. It reads the chunks that the client sends, and keeps them as column batches of 64k rows. Each batch is released when the insert has read it. A load parses a batch in a part per core (CSV with quotes is cut where the quotes before a line end are even), and each core converts its part's fields into the table's columns (Conn.bulkChunks); the load's thread only joins them, sorts the keys and writes the chunk. A chunk is one key run of up to 131,072 rows (CHUNK_ROWS, as DuckDB's row groups of 122,880), and its encoded segment is one WAL frame of at most 64 MiB (MAX_RANGE_BYTES); a segment past it is staged as two runs. Runs of 11K rows (a 16 MiB frame, cut by input bytes) made each scan pay a dictionary, zones and setup per 11K rows: ClickBench at 10M rows ran 0.65 of the time in runs of 127K rows. A server file is refused.
• Passwords authenticate by SCRAM-SHA-256.
• Not supported: COPY BINARY.

Search

Search is not a separate API. It is part of the algebra:

-- full text: rows that match, scored by BM25
SELECT id, title, score() FROM docs WHERE match(body, 'raft consensus') ORDER BY score() DESC LIMIT 10;
-- nearest neighbours under a filter
SELECT id FROM items WHERE tenant = 7 ORDER BY distance(embedding, [0.1, 0.2, ...]) LIMIT 10;

• match reads a full-text index of its column: CREATE INDEX docs_body ON docs USING bm25(body).
• The inverted index answers match, with Block-Max WAND for top-k. It skips blocks whose best possible score cannot enter the current top-k.
• BM25 statistics (document count, average length, document frequencies) are summed over the segments and the delta of the snapshot. Thus scores do not depend on how the data is divided.
• distance with ORDER BY ... LIMIT k becomes a top-k nearest-neighbor source.
  • It uses an exact scan when the filter leaves few rows (cheaper and exact). If not, it uses the vector graph.
  • It applies the filter during the graph walk, not after. Thus a selective filter does not make the result empty.
  • Then it ranks again by exact distance.
• <-> and l2_distance give the Euclidean distance. <=> gives the cosine distance. <#> gives the negative inner product, as in pgvector. The graph keeps squared distances internally.
• A hybrid search is one query: match OR near(distance) in WHERE, ORDER BY rrf() DESC and a LIMIT. BM25 scores and distances are not on one scale, so the ranks are fused (reciprocal rank fusion).

SELECT id, title, rrf() AS score FROM docs
WHERE tenant = 7 AND (match(body, 'raft consensus') OR near(embedding <=> $1))
ORDER BY rrf() DESC LIMIT 10;

• The planner (db/sql/hybrid.giz, called before binding) writes it as two ranked subqueries of the table and a FULL JOIN on the table's key (the hidden rowid for a table with no key). The text side is the match top-k, the vector side the distance top-k; each ranks greatest(50, LIMIT + OFFSET) rows under the other conditions of WHERE. Each select item is computed on both sides and taken from the side that has the row; rrf() in the select list is the fused score. Equal scores keep the order of the key.
• The fused score is rrf(text_rank, vector_rank, k, text_weight, vector_weight): the sum of weight / (k + rank) over the sides that ranked the row (a NULL rank adds nothing). k is search.rrf_k (60), the weights search.text_weight and search.vector_weight (1). A negative or non-numeric setting is an error at the next hybrid search. rrf with three to five arguments is also an ordinary function.
• A hybrid search with GROUP BY, DISTINCT, a window, a set operation, another ORDER BY or no LIMIT is an error that names the form.
• $1 is the query's embedding, or pgml.embed('MODEL', 'raft consensus')::vector in the query (SQL ML §8). A cross-encoder can then order the fused rows again (pgml.rank). The console's Data area writes this query from a form. pgml.tune_search measures the settings against judged queries (SQL ML §8).
• The retrieval metrics of one query compare a ranking with the relevant ids: recall_at_k(ranked, relevant, k), precision_at_k, reciprocal_rank, average_precision (trec_eval's) and ndcg_at_k(ranked, relevant, k, gains) (trec_eval's ndcg_cut; gains is the graded relevance of each relevant id, 1 when it is omitted). The ids are two arrays of one type; k 0 or NULL is all of the ranking. avg over the queries gives MRR and MAP. They and rrf are gizml:ml's, so a program that does not use the database computes the same numbers.

Policies (§1) are predicates added to every scan and search source before execution.

7. Sources

A source is a remote system that a query reads through one typed contract (db/source.giz). Its adapter (PostgreSQL through gizlang:postgres, ODBC through gizlang:odbc, files) reports what it can do (schema, projection, filter, group, join) and returns column pages. The plan gives the adapter what it accepts and computes the rest. The data is as fresh as the remote system. Sources have no common transaction. An adapter never claims a snapshot or a change feed that its system does not give. Following a remote change log (logical replication, binlog) into the local Log is not supported.

Files

Built (db/sql/source.giz, db/core/parquet.giz). With --files DIR, a superuser reads files under DIR as tables, as pg_read_server_files of PostgreSQL permits.

• FROM 'sales/*.parquet' (the format comes from the extension), read_parquet, read_csv(path, delim = '|', header = true) and read_json (newline-delimited), over a glob (*, ?, [..], **).
• The open reads only what fixes the schema: Parquet footers, and the first MiB of a text file. It finds the delimiter, the header and the column types: boolean, integer, float, date, timestamp, or else text.
• A query reads units (a row group, a text file) one wave at a time. It decodes them on every core into the columns that a segment holds, and only the columns that it reads.
• Row groups whose min and max statistics exclude its conditions are skipped.

Parquet support:

Rules:

• Each glob matches a file one time, also with repeated ** segments.
• Execution checks the column names and types of the bound source again, also when a query gives them aliases. Changed CSV headers or field counts, new JSON keys, and incompatible typed values fail. They do not move columns or drop data.
• Missing JSON fields are NULL. An empty string in a text column stays an empty string.

Tests:

• tests/db/sources.giz exercises these contracts, signed 128-bit decimal pages, and malformed or truncated input.
• External plain and Snappy fixtures that DuckDB wrote (200,000 rows each, 12 flat columns) were also checked through the SQL API against the counts and aggregates of DuckDB. This is compatibility evidence for those fixtures. It is not evidence for every possible writer or encoding.

Moving a database into GizDB

gizdb migrate (db/ingest/migrate.giz) copies a StarRocks database (through its FE HTTP SQL API) into GizDB, verifies it with checksums, and runs the same queries on both sides to compare the results. The adapter contract is db/ingest/source.giz: each adapter owns schema translation, snapshot pages, saved progress and checksums. StarRocks is the only adapter. Changes after the copy are not followed.

gizdb migrate copy    --source starrocks://USER@HOST:8030/DB --password-file PATH --target DIR
gizdb migrate verify  --source starrocks://USER@HOST:8030/DB --password-env NAME --target DIR
gizdb migrate compare --source starrocks://USER@HOST:8030/DB --password-file PATH --target DIR --queries QUERY-DIR
gizdb migrate status  --target DIR

• Freeze source writes from the copy until the comparison ends. StarRocks has no global snapshot or change log.
• Give the password as a file or an environment variable name, never in the URL.
• A table without a primary key (a StarRocks DUPLICATE KEY table) is copied by offset in the order of its columns, so equal rows stay equal rows. --key TABLE=COLUMN[,COLUMN] (repeatable) declares a unique key for such a table; the copy is then keyed and fails at the first repeated key.
• --tables a,b selects tables. --where "TABLE=PREDICATE" (repeatable) filters one table. Copy, verification and checksums use the same predicate, which is validated inside a read-only SELECT. Use the same options to resume and to verify. A changed filter or time zone needs a new target.
• Copy and verification send one HTTP request at a time, in keyset pages. --batch N (1 to 1000, default 1000) sets the page size. --timezone ZONE sets the session time zone (default +00:00).
• A copy resumes where it stopped: rows and the cursor commit together. --max-pages N stops after N pages per table.
• Verification reports row counts, SHA-256 checksum sums and the first differing key per table.
• compare runs each .sql file (one read-only SELECT or WITH ... SELECT) on both sides and streams the full result with bounded memory; there is no row cap. -- Unique order: COLUMNS in the file declares an ordered result; other results compare as multisets.
• --json prints one report. Any mismatch or failure exits nonzero. Cutover is manual.

8. Correctness

Oracle tests

Shared SQL queries in tests/db/sql.giz run against SQLite on the same data: tables of 2000 and 40000 rows with NULLs, before and after writes. The rows must agree, in order when the query orders them.

• ASOF queries are compared with correlated SQLite queries for the nearest row.
• Regression and covariance use closed-form linear data, in serial and parallel execution.
• File schema drift, and bounds that are not supported or not valid, assert errors.

SQLite windows:

• SQLite SUM, AVG and TOTAL windows use the accumulator with compensated additions and inverse transitions, in the native phase order of SQLite.
• Approximation, the result storage type and the overflow history survive an empty frame. CURRENT ROW and zero FOLLOWING keep their different transition phases.
• Decimal windows keep the exact decimal extension of GizDB. Ordinary aggregates keep their prefix-overflow and statistical moment contracts.

Mixed columns:

• Temporary heterogeneous expression columns use Column.mixed(Vec<Value>). Each cell keeps its kind through materialization, gather and concatenation. Typed kernels fall back to scalar evaluation.
• Stored segments stay typed columns coerced to the schema, with unchanged file tags. Mixed columns cannot be encoded as segments.
• The reference matrix for inverse windows and heterogeneous columns is connected to the DB scenario.

Generated queries

tests/db/gen.giz uses the method of SQLancer.

• 10,000 typed random queries under a fixed seed agree with the rows of SQLite. They include projections, DISTINCT, ORDER BY ... LIMIT, GROUP BY with HAVING, inner and left joins through an index, set operations, EXISTS, IN, correlated and uncorrelated scalar subqueries, and NULLs in every column.
• Each WHERE p divides into p, NOT p and p IS NULL, whose union must equal the unfiltered rows (ternary logic partitioning).
• Surveys with new seeds (80,000 queries) found no difference.

Crash tests

Built for the log.

• Damaged logs. The log is cut at every byte, every bit of it is flipped, and a zero tail is appended. Recovery must give exactly the complete commits before the damage, or report corruption when complete commits follow it.
• Sync failure. The engine and the log reach files only through the Disk and File interfaces (fs/disk.giz). A test disk fails the first, the second, and each next sync of a workload in turn: log appends, checkpoint files, the directory. Each time, the engine must stop, refuse commits, and reopen with every synced commit, and with the commit in flight complete or absent (tests/fs/fixture.giz).
• kill -9. 1000 loops of SIGKILL during concurrent commits on a real disk lose no acknowledged transaction, and leave no transaction partly applied (bench/db/reliability/crash.giz, result in bench/db/README.md).

9. Text analysis

Built (analyzer version 1):

1. NFKC normalization.
2. Unicode full case folding.
3. Words split at Unicode letters and digits.
4. Hangul, Han, Hiragana and Katakana runs become overlapping grams of two characters. One character alone stays a token. Thus Korean and Japanese match without spaces (the CJK approach of Lucene).

The analyzer version is part of the schema of the column. A change of the version builds the index again. The next analyzer is a Korean morphological analyzer (the model of Kiwi), measured against this one on MIRACL-ko.

10. What GizDB does not do

• A second storage format, log, transaction mechanism or permission path for a new feature.
• Shared-disk cache coherence (RAC).
• Read locks. Readers never wait. Only writes and FOR UPDATE take row locks (§3).
• Its own query language next to SQL. SQL is the outside interface. The plan algebra is the inside interface.
• Silent staleness or approximation. A result states its watermark, and whether a search was approximate (exact is a query option).
• Hot loops moved into C. When Gizlang is slower, the language is made faster.

11. Native versioned journal

The Engine and the WAL carry a versioned journal (db/core/journal.giz): changes() and watches read it, and Kafka producer outcomes (db/core/journalstore.giz) commit with their input.

Positions

• Position(shard, history, sequence, operation) tells a committed transaction apart from its ordered operation.
• COMMIT_END is a completed transaction boundary.
• Positions from different histories cannot be compared. Promotion requires an explicit new snapshot or rebuild.
• Physical WAL offsets stay separate.

Reversible recording

• Reversible recording is active when the engine has a history: a server's primary and Access.local name one before the engine opens.
• Its versioned appendix uses the WAL. The recovery and replication owner replays it.
• Older records replay normally. A historical journal request without reversible images fails with an explicit requirement for a new snapshot boundary.
• A key mutation of a RESP SET without options (Engine.mutate) records only its new row, as journal appendix version 4 does for a put without a before-image. Its earlier version stays in the table (MVCC), not in the WAL a second time.
• A shipped READ COMMITTED UPDATE that sets its columns from its key and constants is a key mutation too (Engine.mutateShipped): no transaction, sealed with the pass's other mutations as one commit. Its rows carry the before-images the statement read, so the record has journal appendix version 2, as the transaction's had. A batch holds rows with images or rows without, never both. A row lock, a unique secondary index or a changed read sends the statement to the transaction path. pg_stat_gizdb.shipped_mutated counts these UPDATEs.
• The legacy scalar changes result keeps its columns. It now refuses a limit that would divide one commit. Thus the next scalar-cursor request does not skip rows.

Producers

admitProducerOutcomeIn keeps a window of 1 to 256 contiguous epoch, sequence and SHA-256 receipts, in the same transaction as the input writes.

• Version-1 metadata loads as a window of one receipt. An established window cannot become smaller silently.
• A new epoch starts at sequence one.
• Native durability cuts do not bypass the selected synchronous replication acknowledgment gate of the server.

Each receipt keeps an optional opaque acceptance result of a maximum of 128 bytes.

• An exact replay returns the result that was published first, also if the caller now computes a different candidate. It does not publish the input again.
• Version-3 metadata carries these outcomes. The Producer and ProducerWindow readers project their original fields.
• Version-1 and version-2 receipts have no result. Retries that depend on an outcome refuse explicitly. They do not invent an outcome.
• A receipt without a result keeps the retry semantics, and keeps the earlier outcomes.
• Results are admitted before the Catalog mutation. The only Codec reader limits declared binary lengths before a slice.
• This generic API does not attach the Kafka producer epoch and sequence on the wire.

Kafka wire adapter

gizdb --dir DATA --port 5432 --kafka-port 9092 --kafka-topic events --kafka-tenant TENANT

This command enables an optional loopback plaintext listener on the authoritative Engine loop.

Configuration and authority

• The default is disabled (kafkaPort: -1). Port zero reports the held ephemeral port through serveReportingPorts.
• Kafka refuses to start with configured passwords, TLS settings or an SQL role catalog: it has no authenticated adapter.
• Native topic tables refuse SQL ACL and policy metadata. SQL guards the reserved _giz_ namespace before grants, ownership and definer views. Superusers stay trusted.
• The enabled listener requires an explicit tenant that is not empty (a maximum of 256 bytes).
• Every native topic operation uses the Catalog scope of that tenant. Table names hash the framed identity of tenant and topic. Topics with the same name in different tenants have separate rows and offset sequences.
• The configured tenant of the listener is trusted startup authority. It is not wire authentication.

Protocol subset

ApiVersions advertises:

• One explicitly configured topic has partition zero. Requests never create topics. Configuration makes the topic transactional (topic format v3, see Reserved offsets).
• Uncompressed magic 0 and 1 messages and magic 2 batches keep NULL and empty binary keys and values as different, and keep timestamps and record headers, in native Engine rows.
• Offsets and topic metadata commit together through Catalog MVCC.
• CRC32 and CRC32C, endian integers and varints use their SDK owners.
• An older Fetch format that cannot keep the stored fields fails explicitly.
• Compressed batches are refused. A producer's control batch is refused. A transactional batch is accepted only in a transactional Produce, and a transactional Produce accepts only transactional batches.
• Consumer group wire APIs, SASL, ListOffsets by timestamp and Kafka replication APIs are absent.
• This is a bounded protocol subset. It is not verified compatibility with Kafka 4 clients.

Idempotent produce

• Nontransactional InitProducerId v0 allocates a durable positive ID, scoped to the tenant, at epoch zero, through the Journal producer receipt and CAS owner.
• Produce 3 accepts one uncompressed idempotent magic2 batch for each partition, with acknowledgments.
• Kafka record sequences advance by the record count. Each canonical Journal batch receipt keeps the wire sequence and count, and the base offset that was accepted first, in its bounded outcome.
• Records, the topic frontier and the receipt publish in one Engine transaction.
• Exact retained retries return that offset without another append, also after a crash and reopen.
• Changed records, sequence gaps, expired receipts, unknown IDs and epochs that are not assigned are refused explicitly.
• Each partition keeps a maximum of 256 batch receipts. No hidden replay occurs outside that window.
• For an idempotent producer, epoch increase, sequence wrap and resumable InitProducerId versions are not implemented. Allocate a new ID before the signed 32-bit record sequence is exhausted.
• A lost initialization reply can leave an unused ID. A retry of InitProducerId safely allocates a new one.
• Modern Fetch builds record batches again without producer coordinates. Keys, values, timestamps and headers are kept. A v3 topic is read as Reserved offsets states.
• An exact Produce retry also uses the configured standby gate with a matching history. Its volatile confirmation resets on restart. A retained offset alone never bypasses that selected durability contract.
• The v0 InitProducerId layouts and wire error codes use the canonical request schema, response schema and Errors owner of Apache Kafka.

Transactional producers

The broker serves the Kafka transactional producer flow on its one node:

• FindCoordinator with key type 1 (a transactional id) names this broker. Key type 0 (a consumer group) gets COORDINATOR_NOT_AVAILABLE (15).
• InitProducerId with a transactional id and its timeout calls Txn.initializeReservedIn: the first call binds a new PID at epoch 0, each next call increments the epoch, aborts the open transaction and fences the old epoch (47). An empty id is INVALID_REQUEST (42).
• AddPartitionsToTxn begins the transaction (Txn.beginIn) when it is not open. Only partition 0 of a transactional topic is accepted. A missing topic, another partition or a topic that is not v3 fails (3), and the other topics of that request get OPERATION_NOT_ATTEMPTED (55).
• Produce v3 with the transactional id stages each batch with Txn.stageReservedIn. The batch gets its offsets at once, and a read-committed Fetch does not show it until EndTxn.
• EndTxn calls Txn.finishIn: it publishes the batches with one commit or abort marker for each topic.
• Each request is one Engine transaction with the usual commit, sync and standby gate.
• The transaction deadline is wall-clock time (System.time()). An expired transaction is not aborted by itself: the next EndTxn abort or InitProducerId does it.
• The Kafka Java client sends Produce with acks=all, which needs a selected standby (syncSlot). Without one the broker refuses it (21), so a single node serves only clients that send acks 0 or 1.

Native transactions

db/kafka/transaction.giz adds, in the Engine, Catalog and Journal: a stable transactional identity scoped to the tenant, a durable binding of PID and epoch, staged batches with reserved offsets (Reserved offsets), and atomic publication or abort.

• initializeReservedIn keeps the PID and increments its epoch on reinitialization. It aborts the open transaction and fences stale callers. Epoch exhaustion refuses explicitly.
• beginIn uses a wall-clock deadline from System.time() that the caller gives.
• stageReservedIn keeps exact retry outcomes in the canonical Journal receipt window.
• finishIn publishes the staged rows, one marker for each topic and the terminal decision in one transaction of the caller.
• Old read snapshots do not change.
• The coordinator has fixed limits: 32 KiB of serialized state, 4096 staged records and 256 batches. It has no parallel WAL, database or retained open Engine transaction.
• The caller owns commit, local sync and the configured standby gate with a matching history. This includes unchanged retries of staging and publication.
• Expired transactions require an explicit abort. No automatic deadline worker is claimed.
• Only the terminal result of the current turn is kept. Retries of older turns refuse. They do not invent offsets.
• A PID bound to a transaction cannot bypass its coordinator through ordinary Produce.

Transactional framing

The only magic2 codec also exposes transactionalBatch, the server-only controlBatch, and bounded typed recordBatches framing.

• Commit and abort markers are returned as metadata with no application records.
• CRC32C, integer framing and varints keep their SDK owners. Row admission checks the batch count before it allocates decoded records.
• Produce accepts transactional input only with its transactional id, and never control input.
• The supported marker layout uses the ControlRecordType and EndTransactionMarker owners of Kafka.

Reserved offsets

The native reserved-offset path uses the explicit Store.upgradeTransactionsIn. It adds one BYTEA column for batch metadata and topic format v3, through the additive Engine schema owner.

• A clean transaction, physical-version credit, a mandatory scan and Catalog revision fences protect the upgrade. Payloads and MVCC versions are never dropped or built again.
• Engine DDL metadata is current, not versioned by snapshot. An old reader sees the additive column. Its v2 topic cut and its original record values stay intact through the canonical empty-fill projection.
• Old writer cuts conflict.

Txn.initializeReservedIn, beginIn, stageReservedIn and finishIn reserve actual offsets together with native rows and the only Journal producer outcomes. Then they publish, in one Engine transaction, bounded row decisions, one commit or abort marker for each topic, and the terminal result of the coordinator.

• The coordinator stores fixed descriptors of 32 bytes, not another copy of the payload.
• Reinitialization aborts outstanding reservations and fences their epoch atomically.
• Limits for each transaction: 4096 data rows, 256 batches, 32 KiB of serialized coordinator and 1 MiB of encoded input. Each topic has a maximum of 256 pending producers.
• Commit and abort retries cannot append another marker.
• Timeout expiry still requires an explicit abort.

Store.fetchVisibleIn walks the shared offset index with row and byte credit. It keeps the identity of batches and producers, and derives the last stable offset from bounded pending frontiers.

• A request inside a batch returns its original containing batch.
• Committed isolation stops at that frontier. It gives explicit aborted producer positions with raw transactional framing, which the consumer must filter. Markers stay typed metadata.
• Uncommitted isolation can return pending input.
• Ordinary Fetch refuses rows that carry batches. It does not discard identity or leak pending input.
• Fetch v4 reads a v3 topic through fetchVisibleIn with the request's isolation.
• Caller commit, local fsync and the acknowledgment of the selected standby with a matching history stay mandatory, also for retained retries.

Fetch, offsets and acknowledgments

• Requests and record payloads have a limit of 1 MiB. Each fetched partition has explicit byte credit and a maximum of 4096 records.
• New native topics have the canonical offset B+tree index.
• Older BLOB topics require the explicit trusted Store.upgradeOffsetIndexIn(engine, tx, tenant, topic, maxVersions, backing), and a commit and sync of the caller, before Fetch. The refusal names this action. It does not silently go back to point probes.
  • The clean caller transaction limits all retained physical versions before the index is built.
  • Additive Engine schema publication keeps the physical columns and every MVCC version.
  • Mandatory table-scan fences refuse concurrent writes. Existing read pins stay valid.
• Fetch walks a maximum of 256 locations for each index batch, overlays the writes of the caller, and refuses missing offsets.
• Fetch 4 encodes one contiguous uncompressed magic2 batch. It preflights the exact frame, varint and header bytes before it builds the batch. Its meaning of timestamps, NULL, binary values and headers does not change.
• Fetch uses the parked client and wakeup owner for minimum bytes and maximum wait. Missing retained offsets fail, without a skip.

Produce acknowledgments:

• They use the local fsync and the receipt gate of the selected standby with a matching history.
• acks=-1 without a selected --sync-slot returns INVALID_REQUIRED_ACKS without a mutation.
• When the selected standby times out, the connection closes with an uncertain outcome. The local transaction can already be durable. Examine it before a retry.
• The Produce request timeout limits the remote-receipt deadline, with the configured synchronous timeout of the server as a maximum. It never selects an asynchronous fallback.
• acks=0 sends no response, and makes no claim of remote durability.
• No new log, consensus, automatic failover or producer receipt ledger is added.

Tests

• tests/db/kafka/run.giz contains the contracts for a real disposable child listener and wire, SQL visibility, crash and reopen, record preservation, CRC, acknowledgment refusal, ownership and Fetch wait.
• The fixture also stages a real acknowledgment of a designated standby, immediate visibility on the receiver, SIGKILL of the primary, crash, reopen and promotion of the receiver, and an uncertain timed-out write with exact checks of local durable reopen.

12. Native storage resources

Logical credits in this section are admission limits. They are not measurements of allocator use or RSS.

Budgeted readers

• It reserves before I/O. It admits the conservative peak of the raw window and the joined payload. It releases live credit on the mutable close.
• Independent views reserve from the same ledger.
• A credit refusal is an explicit I/O status. There is no fallback to a smaller window.
• This primitive is not connected to every Engine and Log caller yet. It does not admit decoded rows, resident tables or transaction operations.

Transaction encoding

The private follow-on for transaction encoding uses canonical EncodedParts of 64 KiB, Readers of operation-offset ranges and the frame CRC owner.

• Commit emits header, operation and Journal pieces, without a join of a full transaction body.
• Journal changes use the canonical bounded encoder before allocation.
• Savepoints keep their encoded cut across chunk boundaries.
• This removes contiguous copies of a commit. It does not remove the residency of the full transaction: offset and own-write metadata, pending pieces and decoded single operations still need shared admission and spill.

Admitted WAL opening

Engine.openOn(disk, repair, Option.some(context)) explicitly admits WAL header and CRC Engine.openWithResources(disk,repair,scope,windowBytes) explicitly admits WAL header and CRC windows, recovery frame headers and replay Readers, through the same actor scope of the caller.

• It carries no Ref in the returned copyable Engine snapshot, and makes no limit or ledger.
• An admission failure keeps its I/O status, closes owned files and releases leases. It never selects the legacy path without admission.
• Inspection stays read-only. A writable reopen still fsyncs before it publishes its recovered cut.
• Legacy open APIs keep their earlier behavior. This WAL-only primitive does not admit the residency of manifests and segments, decoded values or transaction spill.

Scoped temporary pages

Temporary pages also expose explicit APIs scoped to an actor: tempPagesScoped, appendScoped, nextScoped, readScoped, sealScoped, closeScoped, File.readPageScoped and LeasedPage.closeScoped.

• They use the Ref<Resources> of the caller. Admission is stored synchronously before I/O. Finalization loads the same issuer again.
• The serial mut Resources APIs stay available. They share the physical read, CRC, short-write and rollback, and delete implementations.
• Each arena still has one writing owner. A scope can admit independent concurrent work, but copied arena values are not independent writers.
• A failed rollback keeps the temporary credit and poisons the arena. Cleanup releases the credit after the delete.
• These primitives do not spill Engine transaction operations yet.

Journal image sections

Commit keeps the Journal header, the offset directory and the image chunks as different frame sections. It uses the same CRC and WAL queue owners, and does not pack all image bytes into a second chunk set.

• The streaming recovery fixture now enables the recording through the real subscription owner. It verifies all 2,048 binary after-images, operation positions, partial-page cuts and the final commit boundary, before and after reopen.
• Earlier versions of that fixture wrote history but did not enable the recording. Thus they covered only operation replay.

Pending WAL storage

pendingStorage(sys,scope,pageBytes) makes an admitted temporary arena for pending WAL. The actor owns it.

• Engine.commitWithStorage(tx,storage) prepares the same canonical frame as commit. It admits the full scratch frame, then publishes its row and catalog changes and its clock.
• Engine snapshots keep only the identity, the generation and the byte-range coordinates. The resource scope, the file and the arena stay external.
• syncWithStorage(storage) verifies and drains one admitted page at a time to the same WAL, fsyncs it, and advances committed through the same group-commit owner.
• Then the scratch is truncated and its lease is rotated. This fences old aliases.
• A scratch reset failure explicitly reports the cut that is already durable locally, and keeps the temporary credit until the delete.
• Scratch never certifies standby receipt or ACK.

Rules:

• Legacy write, sync, checkpoint and close paths refuse a live stored range, and name the required API that takes a context. A wrong context refuses before I/O.
• Earlier inline pending records drain first.
• A failed staging leaves the new rows and clock not published. An uncertain rollback poisons the context and stops the Engine.
• A WAL, read, CRC or fsync failure does not advance the acknowledged durable cut.
• The log append owner uses the canonical exact-write completion. Thus a short write cannot advance the LSN silently.
• This is opt-in spill of pending WAL only. The SQL and RESP actor adapters do not propagate the context yet. Staged Ops, own-write directories, decoded values and resident Table state stay separate gaps of admission and locality.

Larger than memory

One engine holds every table: segments, the delta, one primary tree and the secondary trees, with the WAL and segment files on disk. Memory bounds it the way a buffer manager bounds pages (Umbra, LeanStore), so a resident read stays a pointer read and data past memory stays usable.

• Budget. An actor's tables keep at most half the memory free when it starts (StorageContext.dataBudget, Engine.dataBudget). Within it nothing changes: no lookup, pin or check besides one tag test on a segment and one on an index page.
• Columns. A segment opens cold: its columns stay in its immutable seg- file. A query reads back only the columns that it names, one block each (ColdHome.column), into cells that every view of the table shares. The cells keep columns while the budget holds them. Past it, a column serves its query and is not kept, and the next Engine.evictCold lets go of the columns least recently read to make room for it: each column of each segment by its own last read (ColdHome.stamps), so the columns of the statements that run now stay. Past the budget, Engine.evictCold also moves the oldest resident segments to their files. Zones, begin stamps and text and vector indexes stay resident. The budget shrinks by what the process lacks: where its room (coldRoom: the room left under GIZLANG_MEMORY_LIMIT, or in the cgroups and on the machine with other processes counted, whichever is smaller) is under 1 / ROOM_RESERVE_SHARE of the budget, the cells keep no new column (ColdMeter.refuses), the engine ticker calls evictCold at once, and it aims lower by the difference. pg_stat_gizdb.cold_held shows the bytes the cells hold. Wide rows reach the budget in few rows, so a check also follows each 1 / EVICT_CHECK_SHARE of the budget in applied operation bytes, a delta holding that much flushes to a segment (Engine.flushWide), and a load run is cut by bytes (Engine.flushRows), not only at FLUSH_ROWS rows.
• Keys. One B+tree a table (Table.primary) maps each key to its newest version, wherever that version is: a cold segment's rows stay in it. A key run (a load's segment, its keys in order) is a leaf of its own key range with no entry a row; a lookup the tree misses halves the runs (Table.locate). A version names the one it replaced (Table.prevAt), so an older snapshot walks back from the newest version (Table.seenAt), as an undo chain.
• Index pages. Still past the budget, the primary and each secondary tree move leaf pages to their own append-only spill- file (Ordered.evictLeaves). The directory keeps a stub with the page's offset, so each copy of a tree reads back the version it names; a write makes the page resident again. Spill files are scratch: open removes them and rebuilds the trees from the segments and WAL.
• Open. A database larger than memory opens: it reads the segment heads, not the columns.
• Large writes. COPY, a large INSERT, and an autocommit UPDATE or DELETE of 100,000 rows or more (Conn.bulkRewrite) run as one chunked transaction: each chunk is one WAL group record applied to the group's copy of the tables, whose full segments go to their files while the load is past the budget. A crash or failure before the commit leaves no row visible; recovery and a standby apply a group only once it is sealed (Engine.replayGroup). A chunk's key check leaves out the load's own chunks, and when every key of the table is in a key run it searches only the runs whose range meets the chunk's (Table.holdsAny). The load's keys are checked against each other once, at the commit (Engine.chunkedDistinct): none when its runs are apart (Table.runsApartFrom), else by their hashes, split by their top bits and sorted on every core. The COMMIT writes each staged file as a chunk of one WAL group at the commit's sequence and attaches the runs (Engine.commitStagedWithContext); it fails with SQLSTATE 40001 when the table or the catalog changed since the COPY. ROLLBACK or the connection's end lets the files go. Any other statement of the transaction first makes the staged rows the transaction's own writes (Conn.materializeStaged). A load's key runs are kept in low-key order (Table.runOrder): a key lookup halves them, then compares a run's rows part by part (Table.runSearch). An INSERT ... ON CONFLICT on the primary key loads the same way (Conn.stageConflicted): a chunk's new keys are one run, and the rows it changes are another, which ends the versions they replace as it attaches (Table.supersede); its WAL chunk carries their before-images (journal appendix version 6), and replay finds those versions again by their keys.
• Queries. A resident scan reads the cold columns that it names after zone maps prune the segments: up to 16 tasks of one thread read the blocks at once (COLD_READERS) and every core decodes them. A scan that would read back more than half the budget fails with SCAN_OVERFLOW, and the statement runs again on the stored path, which streams blocks. A sort that keeps at most 1,024 rows (LIMIT and OFFSET) over a scan reads the columns of its keys and filters first, keeps the top rows by their locations, then reads only those rows whole, a part at a time (lateTop, as DuckDB materializes a top-n late). A cold part decodes only those rows of each column's block (decodeRows: a bit-packed frame, a run, a double, an ALP digit, a dictionary code and its string are found by position). A sort with a LIMIT whose first key is an integer, date or time column of its scan bounds that key first (zoneCut, DuckDB's Top-N dynamic filter): the zone maps' best blocks, about a thousand rows a kept row, set a bound, and the scan reads only the blocks that can pass it. When those rows hold the whole top the sort is done; else it reads four times as many. A scan holds only the parts whose zones may match, so its reads, its kernel's lanes and params and its tasks cost the parts it reads. Sorts, joins, aggregates and windows past their statement credit spill to temporary pages.
• Files. A merge's input segments and the files of an aborted load are removed one checkpoint later (Engine.retired): a thread's view or a cold read may still read them. Open retires the segment files the MANIFEST does not name.

Limits: every key of a resident segment, every begin stamp of an unsettled segment and each tree's inner pages stay in memory; a scan reads a named column of a segment whole, not only the blocks that its zone maps keep. A MANIFEST of the removed persisted-root format (GIZROOT1..GIZROOT3) is refused: recreate that database.

Shared serial and actor Store access

ResourceContext in the Disk owner selects serial Resources or a scoped reference to an issuing actor. It adds no ledger or policy.

• PagedOrdered lookup, rank, cursor and path-copy mutation, and selective column reads of SegmentRef, run one common algorithm with this capability.
• The original serial APIs delegate, and return every changed reservation, also during error cleanup.
• Actor capabilities never keep loaded Resources across I/O. PageCache publishes the admission first, and loads the same issuer again for finalization.
• The mutable cache itself stays under one caller owner. It is not copied independently between concurrent lanes. To serve requests while a miss reads, the owner uses admitContext, runs CacheRead.read in another task and gives its result to completeContext. Each read in flight holds its own slot, so hits and other misses go on. Another miss of the same page, invalidateTail over it and close refuse until the read completes.

Cursors:

• Cursor traversal advances its parent position only after the page admission and read succeed.
• A failed child descent restores the bounded path. Thus, when the memory limit is restored, the cursor continues at the same unconsumed key.
• Close attempts by a foreign issuer keep the live cursor and column contents.

Evidence:

• This includes point, rank and scan, old and new index roots, selective column reads and complete cleanup.
• Logical reservations do not establish allocator or RSS limits, or automatic default SQL spill.

Actor-owned persisted read context

Resource policy

• resourcePolicy(memory,temporary) derives the internal geometry of leases, cache, pages, manifest and queries in the common Store resource owner.
• It accepts the canonical allocator ceiling and explicit temporary and disk credit. Zero temporary credit stays zero.
• resourcesForPolicy refuses edited geometry, and issues through the Resources factory.
• These are ceilings. They are not measured free memory or RSS.
• The next routing step is issuance by the PG actor.

Open with context

• openWithContext admits a bounded format prefix. Then it selects the resident or the persisted-root recovery owner.
• Each owner validates the checksum and framing of its full manifest. A failure never falls back to the other format.
• One StorageContext opens one Engine. A second open refuses before I/O.
• Legacy resident manifests and segments still use their eager decoder. Replay of a stored WAL tail without context stays explicitly not supported.
• Source fixtures cover empty, resident checkpoint, persisted, repeated open and malformed manifests.

Writes with context

putWithContext, deleteWithContext and commitWithContext stage a transaction's rows in the resident tables and its WAL record in the pending storage.

• A commit success is a visibility sequence, not a durability acknowledgment. syncWithStorage must cover it.
• openRecoveryWithContext replays the WAL commit by commit, a sealed chunked group whole (Engine.replayGroup). A read-only inspection writes nothing.
• checkpointWithContext syncs and checkpoints: new segment files, then the MANIFEST, then the logs recovery no longer reads.
• PITR uses the canonical checked record header. A checksummed frame with a truncated flagged header, an invalid sequence or a negative appendix coordinate cannot silently get wall time zero.

Transaction column scans

scanTxColumnsWithContext scans the original registered cut plus the latest own PUTs. It suppresses superseded cold rows and staged DELETEs.

• It projects only the requested physical columns plus primary-key metadata. The canonical Value decoder skips excluded own binary payloads.
• Only StoredTxPoll.done certifies the end. An empty poll that is not terminal is bounded work, and yields.
• Rows and cursors keep their issuer grants until they are explicitly closed.
• A staged mutation or savepoint rollback invalidates a continuation through the canonical Ops revision. Exhaustion refuses before it changes encoded operations, and fences the commit.
• Scope reads record the table-read conflict check.
• Written regressions cover own replacement and deletion, old snapshots, savepoint and serializable conflicts, refused preparation, cold replay, checkpoint reopen, and SIGKILL of a real child process on both sides of group sync.
• Inspection of the prefix and length of an encoded cell is not exposed yet. Callers must not hydrate an excluded payload only to examine its kind. This is a remaining requirement for the inspection of cold containers.

Stored queries

Conn.retainStoredRead binds through the SQL authorization and row-policy owner. Then it gives checked actor acquisitions to the canonical FiniteInput pipeline.

• The cursor requires its original live read-only transaction and the bound catalog.
• The cursor state and each returned delivery have independent physical grants. Callers close deliveries explicitly.
• Conn.statementStored shares the transaction, abort and rollback owner of the ordinary statement, and returns a leased StoredStatement. Its collected result has explicit row and byte limits. Larger results use the finite cursor.
• The initial strategies support one SCAN with a FILTER on rows only, and a PROJECT of columns and constants. Other operators, mutations and session effects return checked errors.

Storage context

storageContext(system, disk, scope, cacheSlots, pendingPageBytes) uses the Ref<Resources> issuer of the caller, and owns one mutable PageCache plus the PendingStorage.

• The Engine stays the logical MVCC snapshot. No issuing reference or cache enters the Engine or read-only worker snapshots.
• getWithContext, scanWithContext and nextWithContext use the same Table traversal and visibility body as their serial versions, through the shared ResourceContext page APIs.
• Do not copy the mutable cache or context of the actor across concurrent lanes and publish it again independently.

Close rules:

• Every returned row and scan must be closed with the owning ResourceContext. A close by a foreign issuer refuses before it drops the live values or cursor.
• Scoped MANIFEST and Log reads load the issuing ledger again at the admission and finalization boundaries. This keeps unrelated claims made during I/O.
• MVCC delta rows are admitted before binary and vector payloads are materialized. They take part in the same Table scan.
• The read-only API does not do stored writes or WAL replay. Those require the explicit writer and recovery methods above.

Stored blocks

The source implementation Table.nextStoredBlockContext accepts selected schema columns, row credit, work credit for invisible rows, and byte credit for prepared output.

• It visits a maximum of one physical column block or delta page for each acquisition.
• It decodes each unique requested immutable column one time, plus a timestamp carrier when necessary.
• Column gather prepares typed vectors directly. Duplicate projections share the source decode. Added columns keep their schema fill.
• Delta text output builds only the selected dictionary, and keeps no unrelated entries.
• Visibility still uses the canonical begin and end interval and the persisted ends index for each candidate row.
• StoredBlock.ids keeps monotone physical source positions, also for invisible candidates. Its scanned counter measures row work.
• Zero visible rows return a bounded block, to yield. Only the physical end returns None.
• Output and decoded-column leases stay with the block until closeContext. Failures leave the scan position unchanged.
• This is conservative admitted credit, not measured RSS.

Prepared SORT input

beginDiskSortInput opens the external SORT strategy for a source that an actor prepared.

• pushDiskSortInput accepts a dense bounded Batch. It checks the actual column metadata, widths, counts and IDs, and the input overlap, before key evaluation or a file mutation.
• Planned type hints do not cast raw or computed cells.
• The same run encoder keeps heterogeneous typed values and canonical equality and order keys, also raw binary, without a new comparator or codec.
• The source order gives the stable tie ordinal. The acquisition actor must keep its authorized plan and context and its MVCC pin, issue batches in that order, and keep and drop their original grants around each push.
• The builder holds no Engine or issuing Ref, and does no legacy table scan.
• finishDiskSortInput uses the merge and seal owners and the LIMIT/OFFSET owners. Then it transfers one ordinary DiskFinite cursor.
• Cursors that are not finished cannot deliver rows. A failed input or merge cannot finish. closeDiskSortInput keeps the cleanup owners.
• Prepared ingest and legacy finite acquisition now share one encoder and merge body.

Mixed aggregate input

DiskMixedAggregateInput sends the dense batches of the original child into every canonical mixed-aggregate descriptor SORT.

• The actor acquires and projects that child one time. Begin, push and finish never acquire it again.
• Its Context APIs use the same issued budget and resource owner for all builders.
• Begin publishes an explicit cleanup slot before builder admission. A rollback failure leaves that poisoned slot with the caller.
• A partial push or finish poisons the acquisition, and keeps both unfinished builders and sealed sources for close.
• Finish transfers sources only after all descriptors seal, in their original descriptor order.
• The bounded descriptor shell has an explicit preflight credit, included in the simultaneous phase and overhead accounting of the execution planner. It stays held until its payload is dropped.
• Foreign and stale contexts refuse before mutation.
• This boundary does not authorize the substitution of a bound context, relax the verification of prepared sources, or certify the MVCC pin of an actor.
• The MODE scenario adds trace receipts of the child that occur one time only, exact rows for each source, and contracts for rollback, partial failure, issuers and cleanup.

Runs

• Rows collect as columns until half the statement's state holds them: their columns (chargedBatch) and 64 bytes for each key and row. A batch past an eighth of the state goes in halves.
• A run's keys are made on the pool, 4,096 rows a task. The keys sort on every core (orderParallel), with the input position as the last key, so the sort is stable. Each column then gathers on a core of its own.
• A run page holds each column's type and its values in the segment block codec (columnBytes in db/core/segment.giz; a mixed column as values, a column nobody reads as nothing), then each row's keys and input position. A page holds at most pageRows rows and about pageBytes. A page past the arena's largest goes in halves.
• The merge keeps a heap of its runs by their heads, and one decoded page of each run. Rows that come from one page in a row go out as one slice of its columns. It merges at most 64 runs a pass, on one core.
• The reader gives slices of a page's columns. OFFSET passes whole stretches of a page. No row becomes values.
• Prepared input admits row and key slots before key evaluation. Single-row layouts that are not possible refuse before a temporary file is acquired.

Prepared GROUP and JOIN input

diskGroupInputPlan and diskJoinInputPlan are the only owners of the prepared sort layout of the operators. Actors can fill those prototypes through DiskSortInput. Then they give one or two sealed cursors to diskGroupedFromSort or diskJoinedFromSort.

• These receivers use the same GROUP allocator (with sparse scalar and statistical states), JOIN admission, and bounded row, probe and finish kernels as finite acquisition.
• They do not capture Env, open another Engine scan, keep an input directory of the population size, or build sorted rows again.
• A successful admission takes the given handle vector. Any refusal leaves the cleanup and the source ownership with its caller.

Admission requires:

• an untouched sealed equality cursor;
• no LIMIT, OFFSET or pending delivery;
• the same budget and page configuration, and the same issuing Resources owner;
• an exact bounded physicalPlanBytes identity with the canonical sort prototype.

Rules:

• This is one conservative admission pass. It is not optimizer equivalence, and not an unbounded plan export.
• Each definition has a limit of min(stateBytes/32, 16 MiB), 16,384 nodes and depth 64. Large inline VALUES definitions can refuse explicitly. Stored-source prototypes do not include their input population.
• Group evaluation inherits the Ctx of the prepared source.
• Prepared JOIN also requires VM sameRowContext: all independent Dialect flags and the scalar bindings of the statement must agree. Plain row contexts with parameters, outer values, subqueries or library bindings stay not supported by that structural check.
• A dialect name alone cannot establish a compatible meaning of expressions.

Statement contexts

• VM issueStatementContext captures the completed Ctx in one managed authority.
• PreparedRowContext carries a plain context or that issued statement. preparedRowContext gets the only expression context.
• Issued contexts agree only when they have the exact same opaque identity. Equal parameter bytes or callable metadata do not establish identity. Function values are never compared.
• beginDiskStatementSortInputContext uses the SORT allocator with this authority.
• Mixed acquisition accepts the same explicit PreparedRowContext. Its prepared receiver also receives the expected context of the actor, and refuses the replacement of one or all source tokens.
• JOIN requires a common source authority.
• Merge, slice, key evaluation, cancellation and recursive restaging keep that same meaning.
• The native issuer owns the lifetime of the authority, and must not overwrite its captured bindings. It stays different from actor authorization and from the pinned MVCC cut.
• Fixtures cover copied bindings, the evaluation of callables, parameters, outer values and subqueries, identities that agree, refusal of substitution, and cleanup.

Scope

• The acquisition actor still owns authorization and the common pinned MVCC cut across sources. A pure DiskFinite carries no Engine transaction token. This API does not certify a snapshot only from equal Ctx values.
• Default Session and PG dispatch stay the responsibility of the actor integration owner.
• The final SORT merge stays synchronous. Bounded output fanout and row cadence come from the admitted operators.
• They also pin refusals of context, prototype, issuer, phase and resource without a loss of ownership.
• Logical grants are separate from RSS.

Close the storage context

1. Close the Engine through closeWithStorage(context.pending).
2. Release its logical snapshots.
3. Close the StorageContext.

• Cleanup refuses open transactions and pending WAL that is not confirmed. It keeps the first cleanup error of PendingStorage.
• This API is not a second reader engine, and not a completed server with bounded memory.

Selected point reads and BLOB prefixes

Selected point acquisition uses Table.getStoredColumnsContext and rowStoredColumnsContext, for resident and stored backing. Full point calls delegate to the same owner.

• The projection attests primary keys and exact MVCC intervals with only the required columns. Unrelated BLOB, vector and dictionary payloads are omitted.

• The actor keeps the registered snapshot and the overlay of staged writes certified by revision. A pure prepared cursor is not an independent pin.

• The same selected-block entry point now accepts resident segments and Builders, without parts() or full-row hydration, with bounded candidate work and a typed output with its own lease.

• Stored BLOB inspection follows the Segment directory and the CRC page cache. It checks encoded offsets, NULL and MVCC framing, and copies only the requested prefix. It bypasses the decoded Column arena and the allocation of the offset vector.

• Physical encoded pages are still read and checksummed, and limited by the stored page limit. This does not claim I/O smaller than a page, or checksum work in O(1).

• Prefix and result leases stay owned by the caller until close.

• Schema fill, an empty BLOB and NULL stay different. Failures publish no projected row or prefix.

Transaction key cursors

The Context transaction API uses one ordered version-key envelope for resident and persisted backing.

• scanTxKeysWithContext accepts selected columns and an optional last logical key.
• nextTxKeyWithContext returns next, bounded candidate work, and a separate complete receipt.
• Deleted or invisible keys can advance next without a row.
• A new transaction seeks after that key under its new registered cut.
• A mutation or savepoint rollback invalidates a transaction cursor.
• The latest own PUTs and DELETEs are merged through the transaction projection of the Engine.
• All returned grants require the original issuer.

Drop and recreate

Engine.dropWithContext stages Op.drop. SQL DROP uses the statement actor admission and the same PreparedCommit, WAL and replay publisher.

• The prepared directory removes dropped incarnations after the row stages finish.
• A DROP and CREATE in the same transaction starts a new physical Table. It never adopts the old payload or index roots.
• No physical files are deleted. Old pure Engine and Table snapshots and their registered cuts keep their original roots.
• Current catalog lookups keep their meaning. Live cursors bound to the catalog invalidate on a catalog change.
• This does not add a separate MVCC catalog for schemas, or a tombstone store.

Catalog revision reads:

• Buffered row, schema and drop mutations record one mandatory read of the catalog revision. Thus mutations across DDL refuse, also at weak isolation, and also for an absent key in a recreated table. Ordinary row concurrency without catalog changes does not change.
• Dropped physical files, and failed files that were not published, stay until a later canonical retention and reclamation implementation proves every relevant cut safe.

Replication receive with context

Replication with context receives bounded raw WAL chunks with Engine.receiveWithContext(sys,lsn,data,storage,policy).

• The transport continues at Log.received. Only the returned Log.end is a durability acknowledgment. It comes after complete-frame CRC validation, canonical replay and a successful fsync.
• The WAL itself holds partial bytes. An incomplete tail refuses local write, sync, rotation, checkpoint, backup and archival. Reopen uses the torn-tail repair and the complete-frame replay owner.
• Wrong transport positions refuse without a mutation. A write, CRC, replay or sync failure stops the receiver.
• A complete frame that was not acknowledged can recover on restart. Do not describe it as rolled back.
• Announced frame bodies and decoded work obey the query budget of the actor.
• CREATE, compatible reshape of retained columns, and DROP and recreation replay use the same canonical physical writer and catalog validator. Type and layout changes that are not supported still refuse before publication.

Empty physical databases and CREATE

A writable openRecoveryWithContext publishes an empty physical ROOT3 cut through the checkpoint publisher, before it returns a database with no tables.

• Its metadata lease stays the authoritative ownership of the physical format by the actor, also before CREATE.
• No additional Engine mode, initialization transaction or WAL is added.
• Catalog objects and the certified commit cut survive an empty legacy migration.

CREATE:

• defineWithContext stages a new schema through the original transaction operation owner.
• Commit prepares durable empty primary, version-end and declared btree secondary roots. Then it uses the row and unique-constraint writer before the original WAL and group-sync publication.
• The same adapter replays CREATE and its initial rows after restart or receive.
• Compatible schemas and DROP and recreation use the adapters above.
• contextTable exposes registered and staged metadata. lockRowsWithContext shares the resident lock and historical-change checks with admitted physical reads.

Evidence:

• Backup with context keeps this empty physical manifest. It does not write a resident one.

Prepared commits

commitWithContext prepares a commit, then publishes it. The prepared commit keeps the physical staging and the actor admission, and it is used one time.

• Retirement closes the stages and clears the payload before it releases the admission.
• A cleanup failure stops the Engine. It never restores write authority.
• Publication checks the clock, catalog, log and root metadata before the pending WAL and the logical publication run. A changed cut refuses before either changes.

Allocation and retained reads

• allocateWithContext, lastChangeWithContext and refreshWithContext use the Table metadata, version-chain and validation owners.
• retainReadWithContext registers the original cut and the staged-read image of a suspended query in the original open-snapshot map.
• The retained read scope cannot refresh or commit the writes of its parent. It closes through ordinary abandon.
• Commits may interleave with an open retained stored read. The caller continues with the StorageContext that retainStoredRead leaves it (readAll, db/access.giz); a copy taken before the retain has a stale page cache, and a commit through it fails with ESTALE (-116).
• Written roots fixtures cover cut retention, preservation of the staged image, allocation exhaustion, preparation refusal and cancellation, and durable publication.

Transaction-visible Context index reads

• lookupTxWithContext(tx,name,index,values,storage,columns,maxKeyBytes,maxStateBytes) makes an equality-prefix cursor, and records the canonical index-entry dependency, also for empty results.
• scanTxIndexWithContext(tx,name,index,storage,columns,spans,descending,maxKeyBytes,maxStateBytes) uses the half-open Span encoding. For ranges, it records a table-scan dependency, to be safe.
• nextTxIndexWithContext returns selected raw candidates, bounded physical work and a separate end. The SQL owner applies predicates, RLS and survivor limits.
• The same APIs serve resident and persisted btree indexes.

The cursor:

• It keeps the original pure Table and MVCC cut, and the transaction operation revision.
• A bounded, admitted projection of the latest own PUTs is sorted one time. Own deletes and replacements suppress committed candidates.
• Physical candidates use the canonical secondary tree and the selected-column decoder. Reverse traversal shares the original paged cursor algorithm.
• This first adapter seeks the secondary tree again for each candidate (O(log N) each). It does not build or scan the relation again.
• Explicit refusals of state, key and row credit keep the unconsumed logical bookmark.
• Mutation, savepoint rollback, closed pins, schema and catalog changes, and foreign issuers refuse continuation.
• Staged row IDs stay -1-opOrdinal at the original transaction revision. Physical IDs keep their canonical locations. Full and projected point reads and binary-prefix reads use that same staged operation location. They do not collapse distinct own rows to one shared -1 identifier.

Evidence:

• Runtime assertions cover both backings, the latest own writes and tombstones, reverse ranges, complete prefixes, conflicts on empty results, old-cut visibility across checkpoint, resource refusal and resume, rollback invalidation and reopen. They have not run yet.

Native bounded WAL groups

operationBatch and Ops.pushBounded admit each PUT and DELETE batch before the original operation codec allocates it. beginChunkedWithContext, stageChunkedWithContext and commitChunkedWithContext keep one registered current MVCC and catalog cut while they prepare bounded batches through the StoredStage owner.

Requirements of this initial native API:

• one drained versioned table;
• reversible Journal enabled;
• a drained, healthy local primary;
• the canonical resource policy of the actor.

DDL, groups across tables, and local publication bound to consensus refuse explicitly. This API does not give arbitrarily large transactions or capacity at TB scale.

WAL layout

• The original WAL contains ordered bounded chunk frames and one final boundary with SHA-256 and count.
• The final boundary is fsynced before the only publication of the Engine root and clock, and before the local durable receipt.
• Intermediate physical frames are not committed cuts, and not replication ACKs.
• Ordinary writer aliases share the managed writer revision of the Log. The group reservation uses it before rollback-handle I/O. Stale copies from before the reservation cannot overwrite or close the current handle.
• A healthy abort that is not sealed restores the original tail.
• A final write or sync failure stops the Engine, and keeps the uncertain evidence for canonical reopen. It is never reported as a successful rollback or as remotely confirmed durability.

Readers

• The same semantic commit dispatcher serves cold reopen, partial receive of Context replication, changes() and WAL-only verification.
• The WAL-only verifier does not give full SQL constraint verification of cold tables, or grouped salvage.

Legacy feed

• The legacy feed in memory keeps a maximum of the queryBytes admission of the native policy and the FEED_KEEP operation cap.
• A group that is too large expires that legacy prefix only at final publication (feedStart advances to the group sequence). Reversible durable Journal images and rows stay intact.
• Consumers receive the explicit trimmed outcome. They do not receive a partial group or a silent Journal skip.

Evidence

• Written contracts cover before and after images of repeated keys, old-cut visibility, abort and reopen, groups larger than the credits of one query, explicit feed expiry, uncertain outcomes after a false fsync, partial receive, and SIGKILL of an owned process at staged, sealed and published cuts.

Query-owned physical reads

Query-owned physical reads use the original Engine getContext, selected point and BLOB, block and transaction cursor methods, with a PageCache and a ResourceContext that the caller owns.

• A query issues its cache under its execution fence. An actor cache cannot get a new label.
• The actor *WithContext methods delegate to these same bodies.
• A cache that does not agree, or is closed, refuses before cursor movement, decoding of own rows or physical I/O.
• Cursors and returned rows close through their original context.
• Checkpoint publication stays the canonical checkpointWithContext owner of the actor. It keeps registered query cuts and immutable physical roots.

Checkpoint plans

ROOT3 checkpoint scheduling uses the original actor publisher planCheckpointWithContext and finishCheckpointWithContext.

• Planning settles the WAL, and admits an immutable manifest for the original root cut, the Engine WAL identity, the actor, the history and the policy. It does not publish, rotate or reset changed rows.
• StoredCheckpointPlan.close cancels or retires that plan. It clears its manifest before it releases the original grant.
• A cancelled, foreign or stale plan refuses, without a change of an admission or conflict into a storage failure.
• checkpointWithContext delegates plan, finish and close, and keeps the original failure.
• Publication still does the original rotation, MANIFEST replacement and directory sync. It keeps live query cuts and never reclaims pinned roots.
• A false sync stops publication without an acknowledgment of the checkpoint. It does not promise a rollback of a MANIFEST that was already written.

Projected transaction blocks

Native projected transaction scans use nextTxBlockContext(tx,scan,cache,context,maxRows,maxWorkRows,maxBytes) with the original registered cut, the selected columns and the own-write revision.

• Each poll acquires one physical Table block, masks keys that the transaction replaced or deleted, and gathers through the Column kernels.
• Physical row IDs do not change. Staged IDs are -1-operationOrdinal, tied to the original transaction revision.
• nextTxBlockWithContext is the actor delegate. Row polls delegate to the same algorithm.
• The returned StoredTxBlockPoll has block, scanned and done. A block with zero rows, or an absent block with done=false, is bounded work. Only done=true certifies completion.
• Every returned block must close under its original Context before its grants are retired.
• Row, work and prepared-byte credit cover selection, gather, key scratch and staged decoding. Decoded physical source columns keep their separate original I/O grants until the block closes.
• Variable equality keys use the only keyValueBounded encoder, admitted before the source key is materialized.
• A failed admission keeps the original cursor position. Foreign or closed query capabilities refuse before I/O or cursor movement.
• Written resident and cold fixtures cover vectorized physical and staged output, exact IDs and rows, bounded empty work, refusal and retry, invalidation by rollback revision, key credits and cleanup.

Physical planning admission

Bounded physical planning calls diskPhysicalAdmission in the SQL executor.

• It shares the original SORT, aggregate, JOIN, set, ASOF, window and recursive admission kernels with allocation.
• The returned minimum is an admission limit, not a prediction of payload size. The final assigned budget must pass the same call before acquisition.
• SORT acquisition uses the original admitDiskSort with its exact equality and prefix layout.
• Mixed aggregation reports descriptor copies and separate acquisition and receiver shells. The graph scheduler owns simultaneous lifetimes.
• Temporary-file metadata and one sequential I/O peak come from tempPagesMemoryCredit of the Store. Caller buffers, held pages, concurrent I/O and the issuer directory stay separate charges under the same Resources.

Capture and plan sizes:

• finiteInputCaptureBytes measures the original retained context, physical plan and LIMIT-stage slots before the graph allocates them.
• Allocation uses the same pipeline walk in construction mode. Measurement never evaluates LIMIT or source expressions.
• Prepared membership keys are visited through the original live Map slots of the SDK, without an allocation of a keys vector.
• This is logical admission credit, not an RSS measurement.
• Plan and expression size admission use the original definition codec in measurement mode. physicalPlanSize counts a checked Rel. expressionSize counts a checked expression vector. Their node, depth and byte limits also apply in write mode.
• Type credit uses typeWireSize of the Value owner, without an allocation of an encoded header.

I/O receipts

Every physical receiver exposes cumulative ioReceipts() from its original temporary-file lineage. This includes merge files that retired successfully, and closed children.

• A transfer gives the receiving owner the same receipts. It does not make another I/O ledger.
• A successful close clears retained plan, context and payload metadata before it releases its grant, and keeps the counters.
• A failed cleanup keeps the original owners for a retry.
• Counts give actual File API transfers and operations. They do not give OS syscalls, device I/O or wire traffic.
• A failed or unobservable direction stays marked unknown. Credit estimates never replace measured I/O.
• The start of destructive cleanup also poisons later polls and source transfer, because a failed close can have cleared kernel state while its original files and grants stay live. Then only a cleanup retry under that issuer is permitted, also with no spare credit.

File planning

A federated FILES source plans through openFilesContext under the statement ResourceContext and FilePlanningLimits.

• CSV layout uses the record grammar of the SDK, with the original delimiter policy of the first 100 records.
• JSON schema inference walks validated value spans and keeps distinct fields, without a full nested JSON AST. Traversal admits its actual retained names and the current recursive stack. The declared match, path and depth limits stay hard work ceilings.
• The canonical compact parser of Parquet measures tree nodes, field IDs and binary spans without a retained tree, before that same parser decodes the footer.
• These are conservative logical credits, not allocator measurements.

Cleanup retries

Context retirement

• Context retirement keeps its original pending arena and manifest grants if the cleanup I/O fails after the cache closes.
• That closed cache prevents more operations. A call of the same StorageContext.close under its original issuer continues the remaining cleanup without resource admission.
• Pending ownership and manifest references clear only after a successful release.
• A Native fixture injects a real refusal of a pending-file close, a repeated refusal, a refusal by a foreign issuer, and a retry with no free memory credit.

WAL writer close

• The shared writer revision of the WAL also keeps a failed file close in a closing state. Append and sync refuse. The original current revision can try the cleanup again.
• An obsolete alias cannot close that handle. Retirement is recorded only after the actual file close succeeds.
• Its fixture keeps the durable WAL cut across refusal and reopen.

Engine close

• Engine.close and closeWithStorage return an explicit I/O error when the WAL file refuses to close. They keep the original Engine for a cleanup retry.
• A closing WAL can be tried again only at a confirmed cut without unpublished writes. Sync errors of a stopped engine have precedence over close errors.
• Pending cleanup separately validates the original leases and the identity of the range that is not confirmed. Thus a scratch reset failure after a successful WAL fsync does not prevent the retirement of that poisoned arena.
• Normal pending reuse still refuses poisoned scratch.
• Pending-WAL fixtures cover both close APIs, an actual reset I/O failure, refusal of a stale generation, refusal of a later write, cleanup with zero free credit, and exact durable rows after reopen.

Restore cleanup

• Legacy restore errors carry RestoreError.cleanup when recovery acquired a WAL owner that refused to close.
• RestoreError.close tries the original Log and scanner again. It never opens a replacement file, and never runs the restore again.
• A successful cleanup clears the carrier. A refusal keeps it.
• The CLI reports the restore error, and reports the cleanup refusal separately.
• Restore with an actor context keeps the cleanup in the original StorageContext.
• The backup fixture injects a real close refusal after recovery, and checks a retry by the same owner without another file open.
• This covers recovery open and retirement. Copy and directory-discovery failures are separate paths. This carrier does not give them a general physical cleanup guarantee.

Opening cleanup

Open failures keep the original WAL cleanup owner separately from the pure LogError and DbError.

• Legacy Log factories return LogOpeningError. Legacy Engine open and inspect factories return OpeningError.
• Their semantic cause does not change. Callers try .close() again on that same error until its physical carrier retires.
• Openings with context transfer it into the StorageContext, with a cleanup slot of the original issuer admitted before the WAL is acquired.
• retryOpeningCleanup() retires the handle and the admission before reuse. A foreign issuer or a refused close keeps both.
• Verification reports and salvage errors keep the same carrier. They do not change it to text.
• Its fixture covers retries of the original handle, refusal by a foreign issuer, cleanup with zero free credit, and exact recovery of scanner admission.
• This contract covers WAL opening owners. Other direct physical opens still require their own original cleanup guarantees.
• Opening cleanup retires separately admitted scanner windows, also when a Log alias has already closed the shared File. A zero writer revision certifies only the retirement of the File. It cannot erase the original resource claims of the scanner. The recovery fixture covers this alias order and exact admission recovery.

Native JOIN fanout credit

The native JOIN fanout credit includes its retained input deltas, the touched State receipts, the peeked pair and the unconsumed output certificate.

• Each transition checks their overlap against the original execution allowance. Enqueue also subtracts output that is already retained in the same page.
• nativeJoinRetainedCredit reports this conservative logical admission. It is not measured RSS.
• Intermediate result pages keep a target that is not published. The coordinator refuses that state before it encodes or admits a checkpoint. Only the completed certificate advances the result frontier.
• JOIN and publication fixtures cover these boundaries.