Operate GizStack storage

Preview Runs today. The interface can still change. Roadmap

This guide is for the operator of a storage range on your own machines. It starts the range on one local disk, mounts it as a file system, and later adds a node with another disk when the first disk becomes full or when you need a second copy. It also tells where each byte is at each step. Use GizStack storage tells how to use buckets, objects, keys and volumes from the command line.

Words used on this page

Before you start

You need:

• Linux with FUSE 3 (fusermount3). Run the node as root to get the kernel's passthrough path. Then reads and writes of an open file go to a local file at local disk speed. Without root, the mount still works, but each read and write is a request to the node.
• One directory on a local disk for each node. Use XFS or ext4. A node keeps its metadata, its chunks and its hot files in this directory.
• Two secrets, the same on every node of the range:
  • GIZ_STORAGE_KEY: the master key, a minimum of 32 bytes. Only node hosts have it. Clients get credentials that are derived from it, never the key itself.
  • GIZ_STORAGE_HISTORY: one name for the range, 1 to 128 bytes.
• A port for each node that another program cannot take first. Choose a port below the ephemeral range of the host (cat /proc/sys/net/ipv4/ip_local_port_range). If that range starts at 1024, outgoing connections can take any port, and a node that starts later can fail with "storage listener unavailable".

Start on one disk

1. Set the secrets, and describe the node. A node descriptor is ID,HOST,PORT,PROVIDER,SITE,INCARNATION. PROVIDER and SITE name the failure domain of the disk: use different values for disks that can fail apart.

   shCopy

   export GIZ_STORAGE_KEY=change-this-master-key-0123456789abcdef
   export GIZ_STORAGE_HISTORY=photos-range-1
   N1="1,10.0.0.1,7401,disk-a,rack-a,1"

2. Start the node, and mount the file system in the same process:

   shCopy

   sudo -E giz storage node start /srv/giz/n1 --id 1 --listen 10.0.0.1:7401 \
       --tenant photos --nodes "$N1" --mount /mnt/giz

   The command serves until the mount is unmounted (fusermount3 -u /mnt/giz). Run it as a systemd service in production. The mount calls the node in the same process, with no socket.

3. On the node host, issue the administrator credential, and configure the client:

   shCopy

   export GIZ_STORAGE_CREDENTIAL=$(giz storage node issue photos 0 0)
   export GIZ_STORAGE_NODES="$N1"

4. Give the root of the file system to the user who writes in it:

   shCopy

   giz storage chown / 1000 1000

giz storage mount MOUNTPOINT DIRECTORY is shorter, but its node listens on 127.0.0.1 only. No other host can reach that node, so that range cannot get more nodes. Use node start with a reachable address for a range that can grow.

Restart a node

Restart a node with the same arguments as its first start. The node writes them to CONFIG in its directory, and refuses a start with other nodes, voters or copies ("storage configuration differs from durable admission"). The node learns later changes of the nodes from the range itself.

After a crash, the new start retires the session of the last run at once, so the first access does not wait for its 30-second lease.

Where the bytes are on one disk

The directory of the node holds:

When a program writes a file through the mount:

1. The kernel writes the hot file of the open file directly (passthrough). The node sees no request.
2. At close or fsync:
   • A file of 1 MiB or less goes to the range as stored writes. They are in the Raft log at once, and become chunks in bytes/ later.
   • A larger file stays in its hot file, now tier/INODE: the hot file is the home of its bytes. The node syncs the file and commits only its size. It does not copy the bytes or hash them. Every other client reads the file from that tier file through the node.
3. The bytes of a tier file go to chunks in bytes/ when another session must change the file, when the mount stops, and before a backup or a change of the nodes.

An fsync returns after the bytes and the commit are on disk. A close does not wait for a sync, as on a local file system.

Add a node

Add a node when the range needs more space, or a second copy of each chunk. A new node takes only the chunks that it owns, and the old nodes then give back the space of the chunks that they no longer own (see Where the bytes are after a node joins).

1. On the new host, set the same GIZ_STORAGE_KEY and GIZ_STORAGE_HISTORY. Then start the node with --join, and give it all nodes: the running ones and the new one.

   shCopy

   N2="2,10.0.0.2,7401,disk-b,rack-b,1"
   giz storage node start /srv/giz/n2 --id 2 --listen 10.0.0.2:7401 \
       --tenant photos --nodes "$N1;$N2" --join

   --join makes the other nodes of --nodes its voters. The node starts as a learner. Keep these arguments for its restarts.

2. On a client with the administrator credential, change the nodes of the range:

   shCopy

   giz storage topology "$N1;$N2"

   The command waits until the change is done, and writes its progress to stderr:

   Copy

   topology: copied 48 of 52 retained roots
   topology: copied 52 of 52 retained roots

   Then set GIZ_STORAGE_NODES="$N1;$N2" on the clients. Clients that run now follow the change.

What topology does:

1. It takes a checkpoint. The new node receives the range's state as one snapshot.
2. It copies each retained root (each chunk that a file, an object or a pin holds) to each node that becomes one of its owners, and checks each copy.
3. It commits the new voters (a joint step, then the final step).

While the change runs, the range does not publish new bytes. Writes wait: a client sends a refused write again for up to 60 seconds, so a program that writes through the mount waits and does not get an error. Reads continue. On the test host, a range of 52 roots (about 33 MiB) took 2.3 s.

If a step makes no progress for 60 seconds, for example because the new node is down, topology cancels the change, and the nodes stay as they were. If you stop topology before it ends, cancel the change yourself:

giz storage topology cancel

A change that has reached its joint step cannot be cancelled. Run the same topology command again, and it continues.

Where the bytes are after a node joins

For example, with --copies 1 and two nodes, each node holds about half of the data after the join and the next scrubs. The split follows the chunk names.

• A join makes space on the old disks. The old node drops a chunk only after the scrub has checked the copy of each owner.
• More copies give more safety, not more space. Each chunk takes copies times its size.

Full disks and lost nodes

The leader asks each node for the free space of its disks every 10 seconds.

• A node whose disks are 90% full or more gets no new chunks. The scrub copies its chunks to other owners and then removes them from the full node.
• A node that does not answer for 60 seconds gets no new chunks either. The scrub makes the missing copies on the other nodes. With --copies 1, the chunks of a lost node are lost.
• When the other nodes cannot hold copies copies in domains domains, the range keeps the node.

More disks in one node

Give a node more directories, each on a disk of its own:

sudo -E giz storage node start /srv/giz/n1 --id 1 --listen 10.0.0.1:7401 \
    --tenant photos --nodes "$N1" --mount /mnt/giz --disks /srv/disk2/giz,/srv/disk3/giz

The node keeps its chunks in bytes/ of each directory. Each disk gets an even share of the chunks, and a write that a full disk refuses goes to the next disk. A disk that you add later moves no chunk. Restart the node with the same --disks. The metadata stays in the first directory.

Back up and restore

giz storage backup /backup/photos

backup writes each chunk that the folder does not hold yet (blob-1-*), then MANIFEST, the signed metadata of one cut. A later backup into the same folder copies only the new chunks. The size of a backup is not limited, except the metadata of the range (32 MiB).

To restore into a new range:

1. Make the directory of the first node, and copy MANIFEST to DIRECTORY/RESTORE and the blob-1-* files to DIRECTORY/bytes/. Do this for each node of the new range.
2. Start the nodes with a new GIZ_STORAGE_KEY and GIZ_STORAGE_RESTORE_KEY set to the key of the source range. The range answers when each chunk is checked.
3. Remove GIZ_STORAGE_RESTORE_KEY before the next restart.

Choose the number of nodes and copies

• Voters. A change commits when most voters have it. With two voters, both must be up: if one stops, the range stops taking writes until it returns. With three voters, one can stop. Use one voter (one disk) or three or more.
• Copies. --copies N keeps each chunk on N nodes. You set it at the first start, and it cannot change later. The range needs at least N nodes.
• Domains. --domains D requires the copies of each chunk to be in D different provider/site pairs. The node descriptors give these names. The range does not check that the names match the real hardware.

A three-node range with two copies in two domains:

giz storage node start /srv/giz/n1 --id 1 --listen 10.0.0.1:7401 --tenant photos \
    --nodes "$N1;$N2;$N3" --copies 2 --domains 2

Start each of the three nodes with its own --id and --listen, and the same --nodes, --copies and --domains.

Remove or replace a node

Give topology the nodes that stay. A node that you leave out stops holding data after its data has moved to the nodes that stay. Then stop it.

giz storage topology "$N1;$N3"

To replace a disk, add the new node first, then remove the old one. A node ID that left the range cannot come back. Give the new node a new ID.

Add a range to the namespace

The metadata of one range is on every node of that range. To grow the metadata past one range, add more ranges to the namespace. Each range is its own group of nodes. All ranges show one file system through the mount of the first range (band 0).

1. Start the nodes of the new range as for a new range. Use their own directories and ports and a new GIZ_STORAGE_HISTORY. Use the same tenant and the same GIZ_STORAGE_KEY as the first range.
2. With GIZ_STORAGE_NODES and an administrator credential of the first range, add the new range as band 1:

giz storage range add 1 "$R1"

$R1 is the node list of the new range. The command gives the new range band 1 and then lists it in the root of band 0. You can run the command again: a band that the range has already taken is no error. A range that took another band, or that already has files, refuses the command.

After this, a new directory at depth 3 or less goes to the range with the fewest inodes. Deeper directories stay in the range of their parent. A mount links each range that the root lists when it starts. It links a range that is added later within 5 s, or when it first meets one of its directories.

In a namespace with more than one range:

• A file or a directory can move to a directory of another range (mv, rename). If a name of another range is already at the target, the rename removes that name first, so for a moment neither file is there. A rename inside one directory is always atomic.
• A file that has more than one name (hard links) cannot move to another range: the rename fails with EXDEV, and mv then copies the file. A hard link between two ranges also fails with EXDEV.
• Directory moves to another parent wait for each other: one client moves a directory at a time in the whole namespace. A client that stops while it holds this lock loses it when its session ends (30 s).
• A mount that stops in the middle of a change across two ranges (mkdir, rmdir, rename, unlink) leaves no error. The next lookup of that name finishes or undoes the change.

Check the range

giz storage usage

usage gives the retained chunks, their bytes (each chunk once), the copy count and the stored bytes (bytes × copies).

The leader checks the copies of each retained chunk every 30 seconds (GIZ_STORAGE_SCRUB_INTERVAL_MS), at most at GIZ_STORAGE_SCRUB_BYTES_PER_SECOND. It repairs a missing or damaged copy from a good copy.

Limits today

• copies and domains are fixed at the first start.
• All nodes of a range hold all metadata of that range. More metadata needs more ranges (Add a range to the namespace). A namespace has at most 31 ranges after band 0. A range does not split or move its directories to another range by itself, and the entries of one directory are always in one range.
• A six-field descriptor uses TCP with signed requests but no encryption: use it on a private network. A nine-field descriptor (ID,HOST,PORT,PROVIDER,SITE,INCARNATION,tls,SERVER-NAME,CA, with the CA certificate in base64) and GIZ_STORAGE_TLS_CHAIN and GIZ_STORAGE_TLS_KEY on each node give TLS between nodes and clients.
• A backup and a change of the nodes wait for the tier files to go to chunks. The range asks the mount to do it, and the command retries.

Troubleshooting

How this page is tested

Tests with real processes run every command on this page except range add: a node joins (node start --join), topology and topology cancel move chunks, two backup runs and a restore, the disks of a node, and two ranges (placement, moves of files and directories between them, half-done changes and the lock of directory moves). After a join, the new node holds chunks and a client of both nodes reads the data.