GizFS: storage internals

GizFS (fs/) keeps files, block volumes and objects in one range of nodes: immutable chunks that a Raft log commits. This page gives the contracts of the code as it is built. To use storage, read the storage guide. To run a range of nodes, read Operate GizStack storage.

Owners

Dependencies

• db/ and the file and object clients depend on the shared fs/ primitives. The shared primitives do not import the database engine.
• The disk and codec owners moved from the database. They were not copied, and the old database paths have no forwarding wrappers.
• The database schema catalog, MVCC and query execution keep their database meaning.

The database seam

The database attaches with the immutable identities and receipts of its native segments and pages, with the shared disk resources and cache admission.

1. A database range prepares its payload separately.
2. It commits that typed payload to the shared range owner.
3. It applies and publishes the accepted history, index and epoch.

• The database WAL does not go through FUSE.
• Engine transactions and file operations each keep their own validation rules.
• A byte receipt is not a database transaction confirmation.
• Journal consumer positions are different from physical log offsets and consensus indexes.

Physical form

Catalog pages

fs/catalog.giz keeps managed metadata and its disk form under one owner. StoredCatalog uses the shared PagedOrdered index, PageCache and ResourceContext.

• An index key holds the canonical binary key and a small descriptor of length and CRC. The value is in a separate immutable file range.
• Sorted unique input streams into a candidate root. Lookup cursors and cursors that resume from a prefix read only the pages that they select.
• A path-copy candidate keeps the previous root. A failed candidate invalidates the cached tail before it truncates.
• The Store and Journal publishers must sync and commit the root that they get back. After a rollback error, writes stop until the tail is validated.
• The codecs of meaningful values stay in the file, object and database clients.
• The page cache keeps the decoded form of an index page with its bytes, under the grant of that page, so a resident page is decoded once. It keeps it only when the cache has room without an eviction. A hit does not change the ledger: the slot counts pins. 20,000 random-key gets: 1,500 → 1,290 ms.
• A path-copy update writes the new pages in one call and puts them in the cache with their decoded nodes. Catalog values go into the cache as they are written. Point, rank and catalog-key reads of a resident decoded page take no pin, grant or cursor (PagedOrdered.residentNode).
• There are two key indexes. A hash catalog (inodes, handles, uploads, keys, object operations) finds a slot by the hash of the key (HashTrie, gizlang:collections), and walks in slot order. The next row reuses a freed slot. An ordered catalog (directory entries, sessions, operations, buckets, object entries and versions, rules, pins) keeps keys in an Ordered and walks in key order (prefix cursors). A native catalog has both: the keys in the PagedOrdered of its pages (root and order), and the hashes in memory. The hashes are built once from the key pages when the catalog is adopted, so a point lookup walks no page. bench/fs/meta.giz (FS.apply only, one directory), create and remove: in memory 50 → 41 µs and 72 → 65 µs; native, 4,000 pages, 123 → 75 µs and 297 → 150 µs.
• A native catalog keeps changes in memory and writes them to its pages in one merge: at most 4,096 rows (NativeRows.pending), at most 64 KiB for each merge (EDIT_BYTES), and at most 512 index edits for each tree (NativeOrdered.pending). The 65,536 rows read or written most recently stay decoded, slot by slot (NativeRows.decoded). A point read looks there first and decodes nothing. Rank reads, cursors and the root of a snapshot (nativeRoot) use it first. The size of a snapshot counts the page root as it is (storedRoot). A command writes no page, because the encoding is the same bytes. The Raft log holds the change until the checkpoint, as it holds pages that are not synced.
• A volume node sizes its caches from a memory budget: GIZ_STORAGE_NETWORK_MEMORY_BYTES, else a quarter of the available memory (Disk.hostPolicy), from 256 MiB to 8 GiB. An eighth is for state pages, a quarter for verified immutable blobs. A blob cache miss hashes the blob once from its source (the local file or a peer copy).
• A read in progress holds four times the page bytes (the read buffer, its copy and a margin). A published page holds only its bytes, 64 bytes and its key (pageCacheResidentCredit). Thus the budget holds the page: 16,384 pages of 4 KiB read again from a cache of twice their bytes take 42 ms instead of 406 ms (hits 0 → 16,384, bench/fs/cache.giz).
• The resource ledger keeps claims in a PageDirectory (no Ref for each step). Range ledgers change in place with Memory.update. 20,000 random-key puts and gets: about 6.8 s and 2.4 s → 5.6 s and 1.7 s.
• Page cache entries hold no Ref, so the collector does not trace cache entries.

The native scenario tests/fs/catalogpages checks 160 MiB of values under a shared resource admission of 512 KiB: binary keys that hold 0 and 255, empty values, resumed cursors, immutable roots, CRC refusal, partial writes, interrupted bulk input, failed rollback, and output grants that stay after a cache eviction.

• File metadata uses this native page form (FS.nativeImage). Object snapshots use the managed catalog and keep the limits of Safety and limits.
• The admission limit is not an operating system RSS limit, and not a capacity measurement of a distributed service.

Spans and chunks

Store.Span is the only physical extent type, validator and binary codec.

• The file system uses it directly. It keeps the inode wire encoding and does not copy the whole extent vector for each read.
• A dense object read uses the original immutable chunk vector. It makes no separate span directory.
• Both use the same bounded path for holes, pieces and digest checks.
• A failed fetch does not change the current byte offset, also for a repeated identity whose access changed.

Store.Chunk holds the pair of an identity and a receipt, and its canonical encoding. Object commands use this physical type directly. Array limits and transaction rules stay with the object owner.

UploadBatch keeps the set of successful receipts for repeated identities in one tenant-scoped upload.

• The number of distinct identities has an explicit limit, and it holds no payload bytes.
• A failed upload is not cached. An empty set, a set that is too large, or a set with the identity of another tenant is refused.
• A cached set gives no authority. Publication still checks the current signature, quorum, policy and placement. A copy still fetches and authorizes each source piece.
• The upload-batch tests check tenant isolation, kept immutable proofs, refusal before an upload over the budget, a retry after a refusal, and wire bytes that do not change.

Manifests

StoredManifest streams sparse or dense spans into the shared StoredCatalog.

• A lookup of the previous entry and an ascending cursor read only the selected extent metadata. There is no second index, page cache or resource ledger.
• A dense manifest refuses a missing range. A sparse hole gives a bounded piece of zeros.
• When a metadata read is refused, its pending address stays. A returned piece holds its own resource grant until a cache eviction and the close of the cursor.
• A descriptor alone gives no right to read bytes and no retention pin. Store and Journal publication and semantic authorization are still necessary.

The native scenario tests/fs/manifest checks 1,500,000 extents and 177,000,000 bytes of value records under a shared resource admission of 512 KiB, over a sparse logical size of 1,572,864,000,000 bytes.

• Small selected reads limit metadata I/O, also for a lookup of the last extent from an empty cache.
• It also checks dense ranges, every interval of a fixture with pieces and holes, bad metadata, failed reads, output admission, the cleanup of old data and a real rollback of the tail.

One local run wrote 177,000,000 bytes of values and 63,388,468 bytes of index in 15,141 ms.

These numbers measure this native scenario. They do not measure a distributed daemon, WAN work or a vendor comparison.

Exact truncation and minimum encodings

File.truncateExact checks the file size after the backend truncate succeeds. Catalog, ordered pages, temporary pages and the Raft tail rollback use this owner, and refuse a false success receipt.

• Raft recovery uses the real minimum entry encoding of 36 bytes, also for an empty election no-op. The conservative log reservation is a separate 40 bytes.
• The Raft scenario checks a restart at an admission bound of exactly 36 bytes. It refuses 35 bytes, a truncate that falsely confirms a torn tail, and a damaged frame followed by a whole frame.

Authorization recovery uses the encoded minimums of an intent (17 bytes) and a rule (33 bytes), with the key and prefix fields that may be empty.

• The authorization scenario checks short fields against the canonical writer, exact snapshot accounting, every cut of the shortest intent, refusal of an empty bucket and the limit of four intents.
• The view of a full bucket gives a bounded head. Object, version, upload and part listings use the paged query owner.

Format versions

Today's code opens storage that any released version wrote.

• Each module that writes a disk format declares formats(): each version (magic) that its reader opens, oldest first. The writer writes the last one (Codec.Format).
• tests/fs/formats/samples/ holds one embedded store for each set of released versions. A sample is named by the hash of its FORMATS file, and SAMPLES holds the hash of each sample file, so a committed sample never changes.
• The suite fails when a declared version has no sample, when a sample holds a version that no reader declares, or when a copy of a sample fails to open, read, write or open again.
• To change a format, add a new version and keep the old reader. Run bin/giz tests/fs/formats/run.giz make DIR, commit DIR with the printed sample folder, and run bin/giz tests/fs/formats/run.giz hashes > tests/fs/formats/SAMPLES.
• A store in a format before GRAFT002 and GRDEL002 is opened once and written again in the current format.

Safety and limits

Nodes and disks

• A service has one tenant namespace and one voter group for each configured volume.
• Node ID, incarnation, provider and site are explicit.
• A participant that only caches or computes is a role that the planner can select. The daemon does not configure it as a durable voter.
• A configured disk backend must be qualified from outside. Calls to fdatasync, rename and directory sync do not prove that a VPS host keeps its flush through a power loss.
• ZFS is optional.

Chunks and receipts

• An immutable chunk is at most 1 MiB.
• A chunk identity holds the tenant, the bytes and the format version. The same bytes of another tenant give no access across tenants.
• A signed receipt binds the chunk, the node incarnation, the placement generation and the policy revision.
• Before publication, the file and object owners check distinct durable nodes, providers and sites. A disposable cache does not count.
• A write copies a chunk to its rendezvous owners among the active nodes. When an owner gives no receipt, the next node in the same order takes it. A local read miss asks the rendezvous owner first, then the other active nodes. A local copy that fails its check is repaired with verified bytes (Store.Local.heal). Scrub restores the owners' copies.
• Damaged duplicate bytes are refused, not read again and not counted. New and repaired bytes are read again and checked before the receipt (Store.Local.publish, Store.Local.repair).
• A failed sync gives no receipt.
• A write of at most FS.storedLimit (Op.store) carries its bytes in the command: 1 MiB on one node, else 64 KiB (FS.STORED_WRITE_BYTES). It has no upload, receipt or staging root. A larger write goes from the client directly to all owners at once (Client.place), and the log commits only identities and receipts: the leader never reads or relays data bytes.
• On a one-node range, the bytes of a write of 64 KiB or more are outside consensus. Before the proposal, the leader writes the bytes once as an immutable blob, without a sync (Volume.logged, Store.Local.keep), and the log entry holds only the identity (file command tag 30). A smaller write keeps its bytes in the entry: one log sync is the cheapest durable write, as the ZFS intent log is. Apply keeps it as a blob, and the checkpoint syncs it. A Raft append carries only entries; with more than one node, an entry never names a blob. Bytes are never written twice to the log, and a checkpoint does not grow with the data.
• Order, as in ext4 ordered mode: before each write that syncs the log, Raft syncs only the new blobs that the entries name (Raft.Node.ordered, syncKept: one Dir.syncFilesystem on the node). With no such blob, it does not sync. Thus a durable entry never names lost bytes. Tag 30 from a client is refused.
• Each node hashes the bytes once: a write of 256 KiB or more on its own thread as the request arrives, before the actor takes it (Volume.arriving); the others in the actor (Volume.resolveStored); a follower hashes the carried bytes on its threads (hashAll). Extents, kept blobs and request digests use that identity (FS.requestDigest names the bytes of a stored write by it). Tags 28 and 29 (entries with bytes) stay, to read old logs.
• A flush sends the small writes that it holds as the pieces of one stored write.
• A node keeps page sums for each local blob larger than 64 KiB (Store.sumName: a CRC32C for each 4 KiB page), written with the blob and not synced. A partial read of such a blob reads its pages and compares them with the sums (Store.Local.readRange): a 4 KiB read reads one page, not a 1 MiB chunk. A smaller blob is read whole. Whole reads, transfers and scrub check the SHA-256 identity. When the sums are missing or torn (a crash, or a store from before the sums), a read reads the whole blob, checks its identity and writes the sums again.
• A read of a mount does not hold the node actor for its bytes. The actor resolves the file and its read pieces, and gives holes and the bytes of blob pages in memory (a plan, Volume.READ_PLAN). The connection reads the other pieces from local pages and checks them (Volume.readPlanned), in parallel with other reads, and also while the actor syncs its log. When a piece is not local or its sum does not agree, the whole read goes to the actor.
• A node also syncs its kept blobs before the log drops their commands (a checkpoint or a snapshot install). Every voter applies each command, so a range needs at least COPIES voters. Volume.Client.writeOp selects an Op.store or an Op.write placement by size. In a placement, the leader gives the generation and the owners with no commit (placeFile, no bytes). The client sends the bytes to all owners at the same time (putFile, Client.place). The leader relays no bytes, the actor does not wait for the transfer, and the write is one commit. In place of a staging pin, an owner leaves a fresh mark beside the blob (fresh-GENERATION-DIGEST), and one directory sync makes both durable. Collection in that generation does not remove a marked blob. A commit refuses receipts of an older generation, so the inventory of the next collection removes old marks (Local.publishFresh, Local.isFresh). The actor only admits a putFile (Volume.PUT_PLAN); the connection hashes and publishes (Volume.putPlanned), so one owner takes many puts in parallel. Removal by collection and trim waits for the puts in progress, and puts wait while it removes (Store.PutGate): a put that arrives between the check and the removal never certifies a removed blob. On 3 nodes and one disk, a sequential write of 64 MiB with fsync went from 1.6-1.7 s to 1.13-1.36 s. When an owner does not answer, the leader placement (uploadFile) selects the next owner in the same order. A client learns the limit of each state (Client.stored). A range that grew past its copy count refuses a larger stored write (EINVAL), and SessionClient.write sends it again as an upload.
• A client asks the node that answered last first, so a write goes straight to the leader. SessionClient.change is the only write path of FUSE, block devices and benchmarks. It learns a new epoch from an ESTALE answer, and does not ask for the state before each write.
• Group commit: the changes that arrive while a commit syncs, and the changes already in the mailbox when a change starts, go into the next commit (Volume.gather). Each change is projected on the changes before it. The leader joins them into one proposal (Raft.Node.proposeMany), and one replication commits it: one log sync on each node, also for a single voter. When exactly those proposals commit next, the leader uses the projected state and results as they are: like a follower, it applies each write once.
• Each node checkpoints its applied state when its log reaches its share of 8,192 entries or 32 MiB. The shares differ, so a quorum never stops at the same time: a follower of rank r among n voters takes (2n+r)/4n, the leader, which all writes wait for, takes 3/4 last (3 voters: 16, 18.7, 21.3 and 24 MiB), and a single node takes 1/2. The rest is the margin of writes that commit while a node cannot checkpoint. A checkpoint syncs the kept byte files (one syncfs) and the page files in parallel.
• An append request carries at most 1 MiB of entries (at least one), so a follower that is behind takes and syncs each part quickly. After a term conflict, a follower asks for the entries after its commit: committed entries agree with the leader, so the leader steps back once, not entry by entry.
• A write gives the root index the blobs of the spans that it changed, and reads the spans that it meets from resident pages. A write of many pieces reads the spans that its pieces meet once, and replaces the stored spans in one merge.

Metadata and history

• A daemon locks its directory exclusively.
• The durable configuration binds the topology, the failure-domain policy and the key digest. An edit of the configuration alone does not change membership.
• Changing metadata is a bounded state machine in memory. It is snapshotted atomically with the Raft cut.
• Raft holds a bounded state envelope. It is not a paged operation WAL.
• A Raft append is one log write and one fsync. The log file grows by 4 MiB of written zeros (File.writeAhead in fs/disk.giz, shared with the database log), so an append overwrites zeros and a sync flushes only data (a 4 KiB dsync write: 1.8 ms over written zeros, 6.2 ms when it grows the file). A reopen reads the log up to the prefix of zero frames and cuts the zeros. The hard state (term, vote, log generation, snapshot) changes, with a directory sync, only when one of them changes. Each log frame records the commit index known when it was written.
• A volume commits every change before it answers. The meaning is the same for any number of nodes, in one process or remote. A fast answer comes from the write delegation of a mount (FUSE). A mount sends a set of changes as one request (Volume.MUTATE_BATCH, at most 128): the volume applies it in one commit and gives a result for each command.
• The bytes of a small write (less than 64 KiB) that has a copy on every node are kept in memory at apply (Volume.keep). Before a checkpoint drops the entry from the log, they are written to a blob file and synced once (writeKept). Thus the file creations (blob and page sums) and directory updates of many small files become one checkpoint. When a node stops in between, the reopen applies the log again and keeps the same bytes.
• A cluster commit does one fsync on each replica, in parallel. The leader sends the append first, and syncs the new entries while the followers sync (Raft.Node.flush). A follower syncs entries before it confirms them, and applies the commits that it learned after the answer (Volume.settle). An append with only a commit (a barrier) is applied before the answer. The leader counts itself in a quorum only with its durable index. The commit goes in the next frame of the leader: an entry is durable on a majority, and a leader that opens again learns the commit again (Raft §10.2.1). A caller that does not flush (the database) gets the leader's sync in the same frame as the commit at the first quorum confirmation. A follower that learns a commit with no new entry appends a frame with no entries, so a follower that opens again has every commit that it confirmed. A node that opens again reads its log up to the last whole frame. A single voter syncs each entry when it proposes, commits the whole log when it opens again, and writes no commit frame. On ZFS, 2,000 appends of a single voter took 44.5 s with 16,018 fdatasync calls before, and 8.2 s with 2,010 after.
• The service refuses, before the proposal, a candidate larger than the snapshot budget that it can recover.
• The result history refuses admission when it reaches its budget.

Thus the admission of metadata and history limits the capacity that a user sees, as disk capacity does. It is not an unbounded global namespace.

Authority

• Every strict read reads the applied state at a cut that a quorum confirmed. A leader that a quorum heard at time t (a read round or a commit) serves reads alone until t + 1.25 s (Volume.quorum). After that, one quorum round confirms it again, and the leader renews before the lease ends. This is safe because a follower that heard a leader within the minimum election timeout (2.5 s) does not vote or start a term (Raft paper §6.4). The other half covers clock-rate drift.
• Every successful change needs a quorum commit in the current term and a deterministic apply.
• A routing cache never stands in for authority. A lost authority gives an error.
• After a timeout, the commit result can be unknown. A native caller keeps the exact operation ID and request bytes. A retry asks for the original result. A changed request with the same ID is refused.

Transport

• A TCP exchange authenticates both directions with a new connection nonce.
• A client keeps up to 16 persistent connections for each node (ENDPOINT_LANES). A call takes the first idle connection, so sequential calls use one connection, and parallel calls (readahead, flush) do not wait in one line. A node keeps one connection for each peer, so calls arrive in order: a call after a barrier sees the barrier's commit applied. A caller waits for its answer on a reply channel, with no polling. A node accepts 1,024 connections.
• A session controller has its own connection, so a renewal never waits behind the session's data calls.
• A client deadline error names the last refusal that it saw.
• Peer keys and tenant UID/GID keys are derived separately. A tenant client cannot forge Raft RPCs or another numeric principal.
• Plain native TCP is an authenticated profile for controlled networks. Confidentiality on a public WAN needs the TLS transport.

The file profile

Namespace and data

The native state machine supports:

• directories, regular files, symbolic links and hard links;
• atomic rename and replace, and unlink with open handles;
• strict writes, append offset allocation, sparse holes, hole punch (punch, cut at the end of the file) and truncate;
• atomic open with truncate, and exclusive create and open;
• compound attributes, permissions, sticky and setgid directories, and owner changes;
• bounded user.* attributes.

Rules:

• Access, modification and change times are Unix nanoseconds. The server stamps automatic times. An explicit past access or modification time stays in the request identity.
• Reads check immutable data.
• A directory listing cookie binds the revision and a lexicographic continuation. When the revision changes, the listing must start again.

Clone

• A directory tree can be cloned atomically under bounded metadata and quota admission.
• A clone keeps its internal hard links and shares the immutable bytes that did not change. A change of the source does not change the clone.
• A clone that is not made by root clears permission bits.
• A clone is a copy of metadata over shared bytes. A constant-time clone is not measured.

Locks and sync

• Native advisory locks tell an opaque process owner, an explicit open-description owner and a whole-file flock owner apart.
• Byte-range conflicts, partial release, PID reports and the explicit release of an owner or a session use the same committed state machine.
• Locks do not block.
• The durability contract. A change through a mount answers before it is durable, as a local file system answers before its journal commits. A crash can lose the changes after the last fsync, but never part of one change and never a change that an fsync covered. fsync, fsyncdir and unmount write the held bytes and make every change of the mount durable (Client.sync). Raft safety does not change: a node syncs its log before it votes and before it confirms an append.
• The changes of other clients (network, block volumes, S3) are durable on a quorum before the answer. Thus a block device flush has nothing to do.

Open files and sessions

• An open, unlinked file keeps its byte root until the last handle closes.
• A successful namespace change collects tombstones that have no owner, and gives back metadata and byte usage.
• An issued session has committed renewal, fencing, confirmation and bounded retirement. When the transport closes, the same owner cleans it up.
• The retained state of an unknown client is recovered only with authenticated authority and the original generation. Automatic detection alone does not prove that collection is safe.

Renewal and expiry use the same comparison of committed generations.

• The loser of a race gets a stale authority (ESTALE).
• An exact retry of an expiry keeps the generation. A fenced session cannot renew.
• The session scenario checks both orders of the race, with no weaker lease or retirement limit.

Replies

The bounded MailboxReply of the protocol belongs to the connection task, outside the deadline child.

• A caller withdraws the reply before it closes the queue. A late actor completion gets false. It is never sent to a slot that was freed or reused.
• A reply takes one completion only.
• The reply scenario checks a timeout abandonment, a completion by the owner and the reuse of a slot.
• An operation that timed out can still commit. A dropped transport answer does not make an unknown result a confirmation.
• A caller must close its reply before the task that owns it ends.

FUSE

The Linux x86-64 FUSE adapter negotiates protocol 7.31. It uses the kernel caches as a local file system does.

• Data cache. File data is in the page cache with kernel readahead, and stays past an open, as on a local file system. Written pages stay too: a read right after a write of 64 MiB takes 8 ms (ext4: 6-7 ms).
  • The changes of another mount are visible close-to-open, as on NFS. At each open, the kernel gets the attributes again, and drops the data of a new size.
  • The mount finds changes of the same size (Adapter.pages). For each file, it keeps the modification time that the kernel saw, and whether it wrote the file after that. When a new time that it did not write, or an entry of the change feed, arrives, it sends a page invalidation, and the next open answers without FOPEN_KEEP_CACHE.
  • FUSE_AUTO_INVAL_DATA is not accepted. Without a writeback cache, the kernel gives a file its own time at a write, so that flag saw the volume's write time and dropped pages that were just written.
• Write delegation and write-back (as CephFS caps and nowsync).
  • A mount takes the write delegation of a directory (FS.Op.delegate; one mount for each directory; profile "delegation" of the lock row). It reserves inode numbers 1,024 at a time (FS.Op.reserve).
  • In a delegated directory, create, mkdir, unlink and rmdir end in the memory of the mount and answer at once. The changes go into a queue in order, and a flusher sends them 64 at a time as one Volume.MUTATE_BATCH. A new inode is made with its reserved number (FS.Op.make).
  • fsync, fsyncdir and unmount wait until the queue is empty, so every change that answered is durable then. A failed change stays as the writeback error of its directory, and the next fsync or FLUSH returns it.
  • When another session reads or changes a delegated directory, the volume gives EAGAIN and a recall notice. The delegated mount gets the notice from the change feed, empties its queue and gives up the delegation. The asker tries again. When the delegation is not given up within 30 seconds, the asker gets an error.
  • The Raft guarantee stays: a volume never answers before the commit. Only the side that answers changed: the mount.
• Write-behind (the work of the page cache of a local file system).
  • A write goes to the adapter, which keeps the bytes and answers.
  • The kept bytes go to the volume in parallel at close, fsync and release, and before a file setattr. When one handle keeps 1 MiB, they start to go in the background. Each handle can have up to 32 MiB in progress, so the volume always has writes to commit.
  • When a handle keeps 256 MiB that it cannot send yet, writes wait.
  • Bytes go in the order of their writes. Bytes that overlap bytes in progress wait for them, so the later write arrives last.
  • Only the bytes that remain go: when two writes overlap, the later one wins, and adjacent writes join into one write for each 1 MiB of the file (Fuse.settled). Writes go in sets (take), and only that set is cleaned up.
  • The mount keeps writes in place, so a write does not copy the extents that the handle holds. The background flush runs in a task that serve owns.
  • A read gets kept and in-progress bytes from the adapter, and the rest from the volume (Fuse.overlay). stat shows the size that this makes. Neither waits for writes.
  • Other mounts see the bytes after close. Data is durable at fsync. A failed kept write is the error of the next write, close or fsync of that handle.
• No kernel writeback cache. With FUSE_WRITEBACK_CACHE, the kernel keeps its own size and modification time for each regular file (fuse_get_cache_mask). It does not take the server's size, so a truncate by another mount does not arrive: a second mount read old bytes after O_TRUNC (fuse.giz INIT_ACCEPTED). An invalidation does not change this, so a global file system cannot use it.
• No disk file on the mount side. A backing file adds a pwrite to each write and a pread to each read (one thread pool round trip each) and gives no durability: a write is durable only on the nodes. RAM write-behind absorbs the same writes with no I/O.
• An open that does not append makes no volume command. The mount keeps the open and reads and writes the file by inode (a write names handle -INODE). The volume checks permissions at each read and at the commit of each write. create makes only the file (when the owner can write it), and O_TRUNC is a size change. An open takes a volume handle only for a lock, an access-time change (once, after a read answers), or before this mount unlinks or replaces the file, so the bytes stay while it is open. A file that another mount removed fails its next read or write with ENOENT, as on NFS. A read-only open sends no volume command, and the blob pages in memory answer the read at once.
• Passthrough (Linux 6.9 and later, a mount of a process that runs as root: a node with storage mount POINT DIRECTORY or node start --mount).
  • A root server has CAP_SYS_ADMIN, which the kernel needs to take a backing file. It is the only path at the speed of a local file system, because reads and writes are not user-space requests.
  • An open regular file is a local file in DIRECTORY/hot (a hot file). The kernel reads and writes it, also through mmap, with no request to the mount (FOPEN_PASSTHROUGH, FUSE_DEV_IOC_BACKING_OPEN on the kept /dev/fuse descriptor).
  • The first open copies the file from the volume (not for a new or truncated file; a file over 4 GiB uses the request path). A read-only open of a file of at most 1 MiB with no hot file does not copy; it reads on the request path, and the kernel page cache keeps the one volume read.
  • Bytes go to the volume when the file is sealed: at each close (other mounts see them after it, close-to-open), at fsync (durable) and at the last close. A seal writes the size and the 4 KiB pages whose CRC-32C changed since the last seal.
  • When the storage can reflink (XFS, btrfs: checked with FICLONE at start), a clone of the hot file is kept as the base after a seal. The next seal reads, with FIEMAP, only the extents no longer shared with the base. On ext4 it reads the whole file. Thus XFS is the recommended hot storage.
  • Closed hot files stay for the next open, up to 8 GiB and 1,024 files in total, while the volume holds the sealed file. A first open copies with eight 1 MiB reads at a time.
  • On a one-node volume, a hot file over 1 MiB is the home of its bytes (a tier file) (Fs.Op.tier, Inode.tier). A mount uses this when the volume has one node at start and the storage has the node's tier directory.
    • close, flush and the first fsync do not seal. The mount takes the byte delegation of the file (Fs.Op.delegate also takes a file), renames the hot file to tier/INODE, and commits only the size and the tier version (Adapter.tierFlush). The queue syncs the storage once before it sends a set that holds a tier: the bytes and the name are durable before the entry that names them. The next fsync syncs only that file. There is no copy, SHA-256 or blob write.
    • A byte change by another session (a write, a truncate, a clone) gets EAGAIN, and the volume recalls the tier (blockedBy). With no open for writing, the mount seals the bytes into extents, ends the tier (Op.tier with size -1) and gives back the delegation (Adapter.untier). Anyone's read is done by the node from the tier file (Store.Local.tierRead, read plan kind 3).
    • Unmount sends every tier file to extents. Backup and node addition refuse with EAGAIN while a tier file stays, and recall it. The node count is checked again each second (Client.topology). With more than one node, a mount makes no new tier, and seals and sends to the replicas.
    • A mount writes its session to MOUNT-SESSION in the storage. After an abnormal stop, the next start retires the old session at once (Client.retireSession): its delegations end without the 30-second lease. Then, while it serves, it sends the remaining tier files to extents (Adapter.recoverTier). Meanwhile, reads come from the tier file and changes wait. A tier file whose tier was not committed is removed: it was not synced.
    • An fsync of a file of at most 1 MiB seals it (a stored write goes into the log). When it is the same as the last seal, it does nothing.
  • A mount with no root (a laptop) uses read and write requests and the write-behind above.
• A request carries at most 1,020 KiB of data (255 pages): a 1 MiB read uses a request and its head. The mount accepts FUSE_ASYNC_DIO, so the direct requests of one file go at the same time.
• Names and attributes are leased to the kernel for 10 seconds (read delegation). The mount asks the volume for the names and inodes that other sessions changed (Client.changes, the last 4,096 changes of the node) and drops them from the kernel (FUSE_NOTIFY_INVAL_ENTRY and _INODE, on a separate stream: a name notice waits for the lock of its directory). With no change, the connection task of the node holds the request until the next change, for at most one second (heldChanges; the actor does not wait). Thus the changes and recalls of other mounts show at once, and a quiet mount sends one request each second. The lease limits what a mount that cannot reach the volume shows. A change through this mount fixes the kernel cache at once.
• Every request runs in parallel (at most 128; open directories are state that the mount shares). One reading task takes requests, and one task for each request handles it to the end and writes the answer itself. /dev/fuse is a stream of the event loop (POLLED): it is read when epoll reports it ready, and it holds no file pool thread. For 50 µs after an answer, it reads the next request without sleeping (readReady, POLL_NS): the next request of a caller that waits for its answer comes within that time, and waking a sleeping thread costs more than the request. When the kernel gives FUSE over io_uring (Linux 6.14 and later, fuse module enable_uring=1), the mount uses it (Uring). It registers URING_DEPTH entries (header and payload buffers) in a queue for each possible CPU. A request comes in the queue of the sender's CPU; the answer is written in the same buffer and committed, with the next request fetched, by one io_uring_enter (COMMIT_AND_FETCH): one system call for each request, and no /dev/fuse read or write. The ring signals an eventfd, which the mount reads as a stream of the event loop (no thread waits in io_uring_enter). Interrupts, forgets and notices stay on /dev/fuse. Without kernel support (before 6.14), only /dev/fuse is used. Only lock waits (SETLKW) and interrupts go through the lock lane. The kernel can have 1,024 background requests (RELEASE, readahead) in progress.
• Each RPC carries the session token, and the volume checks it: a request has no separate authorization round.
• A read answer says whether the read changes the access time (relatime, FS.accessStale). Thus a read is one RPC, and only a stale access time adds an access command.
• A directory is read a page at a time (4,096 entries, an answer of at most 1 MiB), so the adapter has no size limit. Each entry comes with its inode (FS.Listing.inodes), and the mount answers READDIRPLUS: the kernel gets the attributes with the names, so ls -l sends no LOOKUP or GETATTR for each entry. An entry removed after the open can be listed.
• The mount answers a LOOKUP of a name that the kernel forgot (memory pressure, drop_caches). For a directory that missed 8 times with no change, it reads the listing with its inodes, at most 4 pages (Adapter.loadNames), and answers the next LOOKUPs for the lease (10 seconds). A missing name is asked of the volume. A change through this mount, and a change of another session that the change feed shows, drops every listing read (Adapter.forget).
• A missing name is a negative entry (node 0). The kernel keeps it for the lease, as a found name. A create by another mount drops it.
• A command's answer carries the inode that it made or changed (FS.changedInode, Mutation.inode, only when it fits in 16 KiB): CREATE, MKNOD, MKDIR, SYMLINK, LINK and SETATTR are one RPC each, with no stat after.
• LISTXATTR and GETXATTR answer with no RPC from the attributes of the inodes that the mount read (listings, LOOKUP, READDIRPLUS; Adapter.noteInodes, up to 65,536 inodes). ls -l sends a LISTXATTR for each file. A change drops these as it drops the name listings (Adapter.forget).
• The adapter answers ENOTSUP, with no RPC, for an attribute name outside user.*. The kernel first asks for security.capability at each write (FUSE clears SB_NOSEC).

The mount shows:

• open, read, write, create and mkdir;
• rename, link and unlink;
• symbolic links and directory snapshots;
• compound setattr, times and bounded xattrs;
• process locks that do not block, and flock;
• sync.

Limits:

• When this mount holds a lock, FLUSH releases the opaque process owner of the kernel. The last release of a file clears its flock.
• Linux usually queues a FUSE RELEASE asynchronously, so the return of close does not confirm that a remote flock was released.
• SETLKW, which blocks, is refused.
• The Linux FUSE wire does not show an OFD lock profile reliably. Explicit native OFD locks are supported; OFD conformance through a mount is not.
• An operation that is not supported returns ENOSYS or ENOTSUP.

Note: This is not full POSIX conformance.

The mount negotiation and the kernel messages are Gizlang. fusermount3 only grants the OS permission; Linux requires such a helper for a mount without privileges. The runtime adds only the SDK's general descriptor import and the inheritance of child streams. Other kernel and device integrations can use them. An ordinary directory open still refuses devices.

Block volumes

• fs/block.giz uses an issued Volume.SessionClient. Each volume is one sparse file /volumes/NAME. Its fixed size is set when it is made, and block I/O cannot grow it.
• A snapshot clones the file with mode 0400 to /snapshots/NAME@SNAP. A clone of a snapshot or a volume makes a writable volume with mode 0600. Both share the immutable bytes that did not change, through the file clone. The range gives the replicated copies that exist.
• A private staging name holds the file until its size and mode are set. A file link publishes it without a change of the target. A failed or uncertain preparation can leave a hidden staging name. A listing uses the file revision and continuation, and leaves these names out.
• Offsets and lengths must be multiples of 4,096 bytes. One transfer is at most 1 MiB. A flush uses a new file commit barrier. A write already needs a durable quorum commit.
• fs/nbd.giz serves one open volume over a given Stream. It supports the fixed newstyle negotiation, EXPORT_NAME, LIST, INFO, GO and simple READ, WRITE, FLUSH, TRIM and DISC requests. GO tells the block size. TRIM punches a hole with the file punch: the range reads as zeros and holds no extent. An unsupported request flag returns an error.
• The adapter closes the Stream at a disconnect or a failure. The caller owns transport access control, the volume handle and the session. The adapter does not attach a kernel device, and adds no transport authentication or TLS.
• tests/fs/blockvolume checks a 64 MiB sparse volume, random 4 KiB patterns, snapshot and clone isolation, and the confirmed bytes after a SIGKILL of the leader among three local processes. A Gizlang NBD client uses a stream pair. It checks negotiation, reads, writes, flush, discard, disconnect, alignment errors and read-only refusal, and prints the operations per second of sequential and random reads and writes. This is a local measurement, not a WAN result.

Retention and topology

Staging and collection

The lifecycle catalog uses the commit log and the volume snapshot together with file and object metadata.

• An upload commits a staging root before its bytes are checked. A stored write has no upload and no staging root: its bytes come with its command, and the leader and followers sync them as a blob before they sync the entry.
• A collection fence holds the retention roots and increments the publication generation. A receipt that arrives late becomes invalid.
• Collection needs the exact commit generation, term and applied cut.
• Collection is off by default. GIZ_STORAGE_GC_EXCLUSIVE=1 declares that every root of the shared storage is counted. A database that shares the storage must register every root that it committed and prepared before this declaration is true.
• In a remote collection, a follower that does not know the cut yet tries again after one append round.

Scrub and membership

• Scrub checks the owners' copies of each retention root, and repairs a missing or damaged copy from a verified one. Only after every owner copy is verified does it remove the copies on active nodes that are not owners (Volume.trimRoot; the receiving node checks again, with the avoid list that the leader sends, that it is not an owner). It removes nothing during a membership change. A root whose owners cannot be reached, or that lost every copy, is reported as unconfirmed.
• Every 10 seconds, the leader asks the active nodes for their free disk space (statfs, the sum of the disks of --disks) (Volume.probeNodes, in its own task). A node that is 90% full, or that did not answer 6 times in a row (60 seconds), leaves placement: new chunks do not go to it, scrub makes its copies on other owners, and removes the copies of a full node. A node does not leave when the other nodes cannot fill copies and the domains.
• The leader scrubs automatically every GIZ_STORAGE_SCRUB_INTERVAL_MS (default 30 s), at most GIZ_STORAGE_SCRUB_BYTES_PER_SECOND. tests/fs/maintenance checks it.
• A membership plan stops publication, copies the retained chunks to each node that becomes an owner, moves the metadata, then commits the joint and the final membership.
• A plan can be cancelled before the joint membership.
• A retired node identity stays fenced after a restart.
• Client.topology() returns the committed endpoints. A client follows topology changes: when the nodes that it knows do not lead, it gets the node list from the node that answered (Client.learn). The copies of a client share one connection map. giz storage topology NODES changes the nodes and waits until the change ends.

More than one range in one namespace

• Each range has a band of inode and handle numbers (id >> 40). Band 0 is the root. storage range add BAND NODES gives a new store its band (FS.Op.band) and records that range in the root xattr user.giz.range.BAND of band 0. The root holds at most 32 xattrs, so there are at most 31 other ranges.
• A file or directory of another range has its name (a junction) in the range of its directory, and its anchor in its own range. The anchor is string(id) under FS.ANCHOR (number 0, which no inode holds, so no path shows it), and the inode has the directory of the other range as its parent. A name is valid while its inode has that directory as its parent. A lookup confirms a pending name whose parent agrees, and removes a name whose inode moved or is gone (Ranges.resolve).
• A client (a mount) sends a call to the range of the number that it names. A rename across ranges goes in steps (Ranges.move): it makes the old name pending again (with a permission check), takes the new name as pending, changes the parent in the inode's range (FS.renameAnchor, the step that decides), confirms the new name and removes the old one. The next lookup corrects the names that a client that stopped half way left. A rename over a name of another range removes that name first, so it is not atomic. A file with more than one name does not leave its range (an inode has one anchor). A hard link across ranges is EXDEV.
• With more than one range, a directory move to another parent runs under the namespace lock (FS.Op.moving, a lock row of the band 0 root, until the session releases it or ends). Under the lock, the client walks up from the new parent through the ranges to the root. A move into itself is EINVAL. One range sees a path only up to the first directory of another range.
• A new directory at depth 3 or less goes to the range with the fewest inodes. When the inode counts differ by 1/8 or less, the name hash divides them. A deeper directory stays in the range of its parent.
• Not supported: striped directories, and moving an existing subtree to another range in the background.

Erasure-coded stripes

The full chunks (1 MiB) of a cold file stay as erasure-coded pieces (gizlang:erasure) in place of copies (Volume.stripeStep).

• For each scrub page, the leader looks at 64 inodes and collects the full chunks of the files that did not change for GIZ_STORAGE_EC_AFTER_MS (default one day; negative turns it off). A chunk of zeros stays as copies (its parity would repeat).
• The code depends on the number of active nodes and on copies of the policy: 16 or more, LRC(12,2,2) (16 pieces, 1.33 times, one piece repaired from a group of 6, any 3 lost); 10 or more, LRC(6,2,2) (1.67 times); 6 or more, RS(4,2) (1.5 times, 2 lost). With copies 1, there are no stripes. A code must survive at least as many losses as copies does.
• Each piece has one copy on one node (Placement.stripeOwners: rendezvous order, sites take turns). When the policy has more than one domain, one site holds at most as many pieces as the code survives. Thus a policy of two sites does not use LRC (8 pieces on a site).
• For each piece, a pin stripe: + digest (Lifecycle.STRIPE_PIN) holds all pieces of the stripe in code order. A piece finds its stripe with one lookup, and the pin retains it. An administrator cannot release this pin.
• The order of making: read the chunks and compute the parity, commit the pin (so peers accept the pieces), write the pieces to their owners, and only then remove the other copies. When it stops half way, scrub makes the missing pieces again, and removes the other copies only after it confirms that every piece is in place (Volume.stripeWhole, once for each pair of leader and owners).
• A piece that no node holds is made again from the other pieces (Volume.rebuildPiece). Reads and scrub use the same path: first the group (6 pieces for LRC), else the global parity. A node that takes a TRIM checks ownership as a stripe owner too.
• When no file or object refers to half or more of the chunks, the stripe is dissolved (Volume.dissolveStep): the chunks still in use first get the copies of the policy, then the pin is released. Collection takes the rest.
• Measured (bench/fs/erasure.giz): LRC(12,2,2) encodes 5.5 GiB/s of data on one core; one piece is repaired 2,500 times a second from its group (6 pieces), and 1,600 times a second from the global parity (12 pieces).
• Tests: tests/fs/stripes (6 processes, copies 3, RS(4,2): 12 copies of 4 chunks become 6 pieces; reads and scrub make a lost piece again; the stripe of a removed file is dissolved and collection takes all of it), tests/sdk/erasure and tests/fs/placement.test.giz.

Backup and restore

• A signed backup takes the file, object, multipart, lifecycle and operation-result state at one commit cut, and retains its byte closure.
• A restore needs an empty history and a new, different credential. Reads and writes are not available until the committed byte range is repaired.
• A backup bundle (format 3) holds the metadata image and the names of the chunks only. The chunks go as separate files: giz storage backup DIR (Client.backupTo) writes to DIR only the chunks that DIR does not have, as blob-1-*, and writes MANIFEST last. Another backup to the same DIR copies only new chunks (an incremental backup). The only size limit is 32 MiB of metadata (the volume snapshot limit). Format 2 bundles (which hold the chunk bytes) still restore.
• A restore of format 3 reads the chunks one at a time from the node's bytes/ and checks their identities: copy MANIFEST to RESTORE, and the chunk files to bytes/.
• A restoring node (fs/server.giz) reads RESTORE with GIZ_STORAGE_RESTORE_KEY and a new GIZ_STORAGE_KEY. An embedded store restores with Node.restoreEmbedded (the storage guide). A normal restart removes the restore key.
• A learner uses the voters that are committed now as GIZ_STORAGE_VOTERS.

The object and document profile

Versions and protection

The object catalog uses the same immutable chunks and the same committed volume journal. A bucket belongs to the tenant of the range.

• Versioning is always on. An overwrite keeps the previous version. A normal delete makes a marker. An explicit version delete keeps every other version.
• A metadata search is a bounded scan with exact conditions, through the same authorization owner. It is not a separate index.
• Legal hold and COMPLIANCE retention apply to each version. A tenant authority can release a hold explicitly. A retention date cannot become earlier.
• A marker or an overwrite does not remove held bytes.
• A copy makes a target version with its own protection. The protection of the source still applies to the retained source.
• A volume stamps automatic object times as they arrive. This time does not change the request digest. A retention date stays in the digest.
• Each object result records its request time. The Raft apply removes results older than the 7-day signature window by the committed request time. An expired lookup is refused.

Note: These are application invariants. This is not a regulatory certification or a full ECM workflow product.

Named refs

A CMS, an ECM or a drive keeps draft, review and published states as named refs.

• A ref is an ordinary object KEY/.refs/NAME. Its body is the decimal version id of the KEY that it points to. The versions of the ref object are the history of the ref.
• Client.moveRef(bucket, key, name, from, to) moves a ref from version from (0: no ref yet) to to. It compares the ETag of the ref with from, and puts to on the condition that the ref object version did not change.
• The result is RefMove.moved(version) (the new ref object version) or RefMove.conflict(). Of two moves from the same version, one moves and one gets a conflict (tests/fs/cli/run.giz).
• The caller names a version of a KEY that exists. A ref name has no /.
• Client.ref(bucket, key, name) gives the version that a ref points to (0: no ref).
• Other object helpers of a program: Client.put(bucket, key, data, contentType) puts data as the new latest version and gives that version. Client.putFrom(bucket, key, size, contentType, piece, done) reads PART at a time with piece(offset, count), so memory holds one part. Past PART it is a multipart upload, and parts that agree with an unfinished upload of the same key are not sent again. Client.get(bucket, key, version) reads a version (0: the latest), checks the identity of each chunk, and releases its read pin on every path. Client.rows(query, most) reads every page of a listing, and fails past most rows. giz storage uses the same helpers (fs/cli.giz).
• The CMS keeps the draft, review and published refs of a page (the web internals).

Cold files (object tier)

objectDisk(client, bucket, prefix) of fs/tier.giz is a Disk that shows the objects of a bucket as files. A store such as a database keeps hot files on its own disk, and moves them to the bucket with Disk.replaceRange when they are cold. A reader uses the same Disk, File and PageCache as before.

• open pins the latest version. A later version or a collection does not change what the open file reads. close releases the pin.
• readAt fetches only the chunks that it touches, and checks the identity of each (Client.fetchChecked). A file holds its last 4 chunks, so the pages of one chunk are fetched once. A PageCache keeps the file open.
• A file is written only whole (replaceFile, replaceRange, Client.putFrom). A new version is put, then the previous one is removed. Other writes are READ_ONLY (EROFS). A put is durable when it returns, so sync has nothing to do.
• remove removes the current version. A listing of names goes one level at a time (separator /).
• Tests: tests/fs/tier.test.giz (two range reads of a 3 MB file fetch 3 chunks; 8 PageCache pages fetch 1 chunk).

One range in more than one region

A region is the site of a node. One range spans more than one site, with no region-aware layer above it.

• A read of a client with a site (GIZ_STORAGE_SITE, Client.inSite) goes first to a node of the same site. That node answers when it applied up to the position that the client saw and heard the leader within one second; else the leader answers. A client always sees its own writes; the writes of others can be up to about one second late.
• A node reads a chunk from a copy in the same site first. A node that does not own a chunk keeps the copy that it read from a peer in its cache (cache/, Store.BlobCache), so a second read is local also in a site with no copy. The cache is a tenth of the free space (at most 64 GiB, GIZ_STORAGE_CACHE_BYTES), and removes the oldest copies first. A cache copy is not a durable copy and gives no receipt. Each read checks its identity, and when collection removes a blob, the cache copy goes too.
• The leader counts writes by site. When one site makes two thirds of the writes for one minute and the leader stayed for five minutes, it hands leadership to a voter of that site that caught up. When that site has no voter, or the leader's site is not a majority of voters, a node of that site becomes a voter in place of a voter of another site.
• Tests: tests/fs/placement.test.giz (site decisions), tests/fs/cluster (near reads and refusals) and tests/fs/store (the cache).

The S3 profile

fs/s3.giz implements a selected profile of the S3 protocol. It does not claim full AWS compatibility. The HTTP(S) gateway is an adapter to the same committed range, and uses one idempotency ledger. It has no separate byte store or multipart coordinator.

Bodies and budgets:

• Buffered convenience requests, XML and control bodies keep an admission of 16 MiB.
• The gateway streams PUT, UploadPart, GET and Range to the shared immutable byte owner. It never joins an object into one buffer.
• Object bytes are admitted as the checked sum of immutable identities under the chunk budget of the range, not as a whole HTTP body buffer.
• A chunk is at most 1 MiB. A whole snapshot of the object catalog, with retained versions and staged multipart manifests, keeps an exact encoded budget of 8 MiB.
• The catalog, count and command budgets apply too.

Reads and uploads:

• A read takes a committed pin and reads the version that it pinned. The last byte of a whole read is held until the whole digest passes.
• A failed or unfinished upload publishes no catalog version. Staged bytes stay under the shared lifecycle fence.
• A SHA256 ETag is an opaque validator. It does not imitate the MD5 or multipart composite ETag of AWS.
• The completion XML grammar is bounded. Unknown elements, a DTD and external entities are refused.
• UploadPartCopy can send the bytes again. It does not claim zero network movement.

These parts of the profile are not supported, and return an explicit error:

• STS;
• ACL and IAM policy APIs;
• S3 lifecycle and replication configuration;
• GOVERNANCE bypass;
• SSE and KMS;
• storage-class policy;
• multipart trailer signatures;
• unknown checksum algorithms;
• multiple ranges.

Listings:

• Object, version and multipart upload listings use the same catalog grouping rules.
• Each common prefix uses one result. Its continuation skips that whole encoded index interval, with every past version, delete marker and upload ID.
• A resume after a group needs no scan of the directory size and no second index.

Tests

Run the suite of the area that you changed. CI runs all of them.

bin/giz test fs/SUITE
bin/giz test sdk/http

The cluster and S3 wire suites also cover live replacement, a signed restore, and a restart of every node during a multipart upload.

Planner

The planner uses one constrained objective function with directed egress, measured capacity, and the costs of compute and data transfer.

• It selects each role separately.
• It needs a benefit larger than the cost of a move, a minimum stay, hysteresis and a budget.
• It makes plans. It does not provision servers and does not weaken safety constraints.