# Operate GizStack storage

:::status{of="storage"}
:::

This guide is for the operator of a storage range on your own machines. It starts the range on
one local disk, mounts it as a file system, and later adds a node with another disk when the
first disk becomes full or when you need a second copy. It also tells where each byte is at each
step. [Use GizStack storage](../guides/storage.md) tells how to use buckets, objects, keys and
volumes from the command line.

## Words used on this page

| Word | Meaning |
| --- | --- |
| Range | One storage system: its files, objects, keys and history. All nodes of a range hold the same metadata. |
| Node | One `giz storage node start` process with its own directory on its own disk. |
| Voter | A node that votes on each change of the range (Raft). A change commits when most voters have it on disk. |
| Learner | A new node that receives the range's state and bytes but does not vote yet. |
| Chunk | An immutable piece of file or object bytes, named by its SHA-256. Chunks are the unit of placement. |
| Copies | How many nodes keep each chunk. You set it when you create the range. |
| Hot file | The local file that the kernel reads and writes for an open file of the mount. |

## Before you start

You need:

- Linux with FUSE 3 (`fusermount3`). Run the node as root to get the kernel's passthrough path.
  Then reads and writes of an open file go to a local file at local disk speed. Without root, the
  mount still works, but each read and write is a request to the node.
- One directory on a local disk for each node. Use XFS or ext4. A node keeps its metadata, its
  chunks and its hot files in this directory.
- Two secrets, the same on every node of the range:
  - `GIZ_STORAGE_KEY`: the master key, a minimum of 32 bytes. Only node hosts have it. Clients
    get credentials that are derived from it, never the key itself.
  - `GIZ_STORAGE_HISTORY`: one name for the range, 1 to 128 bytes.
- A port for each node that another program cannot take first. Choose a port below the
  ephemeral range of the host (`cat /proc/sys/net/ipv4/ip_local_port_range`). If that range
  starts at 1024, outgoing connections can take any port, and a node that starts later can fail
  with "storage listener unavailable".

## Start on one disk

1. Set the secrets, and describe the node. A node descriptor is
   `ID,HOST,PORT,PROVIDER,SITE,INCARNATION`. PROVIDER and SITE name the failure domain of the
   disk: use different values for disks that can fail apart.

   ```sh
   export GIZ_STORAGE_KEY=change-this-master-key-0123456789abcdef
   export GIZ_STORAGE_HISTORY=photos-range-1
   N1="1,10.0.0.1,7401,disk-a,rack-a,1"
   ```

2. Start the node, and mount the file system in the same process:

   ```sh
   sudo -E giz storage node start /srv/giz/n1 --id 1 --listen 10.0.0.1:7401 \
       --tenant photos --nodes "$N1" --mount /mnt/giz
   ```

   The command serves until the mount is unmounted (`fusermount3 -u /mnt/giz`). Run it as a
   systemd service in production. The mount calls the node in the same process, with no socket.

3. On the node host, issue the administrator credential, and configure the client:

   ```sh
   export GIZ_STORAGE_CREDENTIAL=$(giz storage node issue photos 0 0)
   export GIZ_STORAGE_NODES="$N1"
   ```

4. Give the root of the file system to the user who writes in it:

   ```sh
   giz storage chown / 1000 1000
   ```

`giz storage mount MOUNTPOINT DIRECTORY` is shorter, but its node listens on 127.0.0.1 only.
No other host can reach that node, so that range cannot get more nodes. Use `node start` with a
reachable address for a range that can grow.

### Restart a node

Restart a node with the same arguments as its first start. The node writes them to `CONFIG` in its
directory, and refuses a start with other nodes, voters or copies ("storage configuration differs
from durable admission"). The node learns later changes of the nodes from the range itself.

After a crash, the new start retires the session of the last run at once, so the first access does
not wait for its 30-second lease.

## Where the bytes are on one disk

The directory of the node holds:

| Path | Holds |
| --- | --- |
| `metadata/` | The Raft log and snapshots: names, inodes, extents, objects, keys. |
| `bytes/` | The chunks (`blob-*`), each written once and never changed. |
| `tier/` | Hot files that are the home of their bytes (see below). |
| `hot/` | Hot files that are copies of committed chunks. |
| `CONFIG`, `MOUNT-SESSION` | The first-start arguments, and the session of the mount's last run. |

When a program writes a file through the mount:

1. The kernel writes the hot file of the open file directly (passthrough). The node sees no request.
2. At close or fsync:
   - A file of 1 MiB or less goes to the range as stored writes. They are in the Raft log at once,
     and become chunks in `bytes/` later.
   - A larger file stays in its hot file, now `tier/INODE`: the hot file is the home of its bytes.
     The node syncs the file and commits only its size. It does not copy the bytes or hash them.
     Every other client reads the file from that tier file through the node.
3. The bytes of a tier file go to chunks in `bytes/` when another session must change the file,
   when the mount stops, and before a backup or a change of the nodes.

An fsync returns after the bytes and the commit are on disk. A close does not wait for a sync, as
on a local file system.

## Add a node

Add a node when the range needs more space, or a second copy of each chunk. A new node takes only
the chunks that it owns, and the old nodes then give back the space of the chunks that they no
longer own (see [Where the bytes are after a node joins](#where-the-bytes-are-after-a-node-joins)).

1. On the new host, set the same `GIZ_STORAGE_KEY` and `GIZ_STORAGE_HISTORY`. Then start the node
   with `--join`, and give it all nodes: the running ones and the new one.

   ```sh
   N2="2,10.0.0.2,7401,disk-b,rack-b,1"
   giz storage node start /srv/giz/n2 --id 2 --listen 10.0.0.2:7401 \
       --tenant photos --nodes "$N1;$N2" --join
   ```

   `--join` makes the other nodes of `--nodes` its voters. The node starts as a learner. Keep these
   arguments for its restarts.

2. On a client with the administrator credential, change the nodes of the range:

   ```sh
   giz storage topology "$N1;$N2"
   ```

   The command waits until the change is done, and writes its progress to stderr:

   ```
   topology: copied 48 of 52 retained roots
   topology: copied 52 of 52 retained roots
   ```

   Then set `GIZ_STORAGE_NODES="$N1;$N2"` on the clients. Clients that run now follow the change.

What `topology` does:

1. It takes a checkpoint. The new node receives the range's state as one snapshot.
2. It copies each retained root (each chunk that a file, an object or a pin holds) to each node
   that becomes one of its owners, and checks each copy.
3. It commits the new voters (a joint step, then the final step).

While the change runs, the range does not publish new bytes. Writes wait: a client sends a refused
write again for up to 60 seconds, so a program that writes through the mount waits and does not
get an error. Reads continue. On the test host, a range of 52 roots (about 33 MiB) took 2.3 s.

If a step makes no progress for 60 seconds, for example because the new node is down,
`topology` cancels the change, and the nodes stay as they were. If you stop `topology` before it
ends, cancel the change yourself:

```sh
giz storage topology cancel
```

A change that has reached its joint step cannot be cancelled. Run the same `topology` command
again, and it continues.

## Where the bytes are after a node joins

| Data | Where |
| --- | --- |
| Metadata (names, inodes, objects, keys) | On every node: each voter keeps the full Raft log. |
| Chunks that existed before the join | On their owners in the new list of nodes. The new node gets the chunks that it owns. Then the scrub removes the copies on the nodes that are no longer owners. |
| Chunks written after the join | On `copies` nodes. The range chooses them for each chunk by rendezvous hashing of the chunk's name. |
| Tier files | Not used. With more than one node, a close or fsync of a large file writes its changed pages as chunks to the chosen nodes. |

For example, with `--copies 1` and two nodes, each node holds about half of the data after the
join and the next scrubs. The split follows the chunk names.

- **A join makes space on the old disks.** The old node drops a chunk only after the scrub has
  checked the copy of each owner.
- **More copies give more safety, not more space.** Each chunk takes `copies` times its size.

## Full disks and lost nodes

The leader asks each node for the free space of its disks every 10 seconds.

- A node whose disks are 90% full or more gets no new chunks. The scrub copies its chunks to other
  owners and then removes them from the full node.
- A node that does not answer for 60 seconds gets no new chunks either. The scrub makes the
  missing copies on the other nodes. With `--copies 1`, the chunks of a lost node are lost.
- When the other nodes cannot hold `copies` copies in `domains` domains, the range keeps the node.

## More disks in one node

Give a node more directories, each on a disk of its own:

```sh
sudo -E giz storage node start /srv/giz/n1 --id 1 --listen 10.0.0.1:7401 \
    --tenant photos --nodes "$N1" --mount /mnt/giz --disks /srv/disk2/giz,/srv/disk3/giz
```

The node keeps its chunks in `bytes/` of each directory. Each disk gets an even share of the chunks,
and a write that a full disk refuses goes to the next disk. A disk that you add later moves no
chunk. Restart the node with the same `--disks`. The metadata stays in the first directory.

## Back up and restore

```sh
giz storage backup /backup/photos
```

`backup` writes each chunk that the folder does not hold yet (`blob-1-*`), then `MANIFEST`, the
signed metadata of one cut. A later backup into the same folder copies only the new chunks. The
size of a backup is not limited, except the metadata of the range (32 MiB).

To restore into a new range:

1. Make the directory of the first node, and copy `MANIFEST` to `DIRECTORY/RESTORE` and the
   `blob-1-*` files to `DIRECTORY/bytes/`. Do this for each node of the new range.
2. Start the nodes with a new `GIZ_STORAGE_KEY` and `GIZ_STORAGE_RESTORE_KEY` set to the key of
   the source range. The range answers when each chunk is checked.
3. Remove `GIZ_STORAGE_RESTORE_KEY` before the next restart.

## Choose the number of nodes and copies

- **Voters.** A change commits when most voters have it. With two voters, both must be up: if one
  stops, the range stops taking writes until it returns. With three voters, one can stop. Use one
  voter (one disk) or three or more.
- **Copies.** `--copies N` keeps each chunk on N nodes. You set it at the first start, and it
  cannot change later. The range needs at least N nodes.
- **Domains.** `--domains D` requires the copies of each chunk to be in D different
  provider/site pairs. The node descriptors give these names. The range does not check that the
  names match the real hardware.

A three-node range with two copies in two domains:

```sh
giz storage node start /srv/giz/n1 --id 1 --listen 10.0.0.1:7401 --tenant photos \
    --nodes "$N1;$N2;$N3" --copies 2 --domains 2
```

Start each of the three nodes with its own `--id` and `--listen`, and the same `--nodes`,
`--copies` and `--domains`.

## Remove or replace a node

Give `topology` the nodes that stay. A node that you leave out stops holding data after its data
has moved to the nodes that stay. Then stop it.

```sh
giz storage topology "$N1;$N3"
```

To replace a disk, add the new node first, then remove the old one. A node ID that left the range
cannot come back. Give the new node a new ID.

## Add a range to the namespace

The metadata of one range is on every node of that range. To grow the metadata past one range,
add more ranges to the namespace. Each range is its own group of nodes. All ranges show one file
system through the mount of the first range (band 0).

1. Start the nodes of the new range as for a new range. Use their own directories and ports and a new
   `GIZ_STORAGE_HISTORY`. Use the same tenant and the same `GIZ_STORAGE_KEY` as the first range.
2. With `GIZ_STORAGE_NODES` and an administrator credential of the first range, add the new range as
   band 1:

```sh
giz storage range add 1 "$R1"
```

`$R1` is the node list of the new range. The command gives the new range band 1 and then lists it in
the root of band 0. You can run the command again: a band that the range has already taken is no
error. A range that took another band, or that already has files, refuses the command.

After this, a new directory at depth 3 or less goes to the range with the fewest inodes. Deeper
directories stay in the range of their parent. A mount links each range that the root lists when it
starts. It links a range that is added later within 5 s, or when it first meets one of its directories.

In a namespace with more than one range:

- A file or a directory can move to a directory of another range (`mv`, `rename`). If a name of
  another range is already at the target, the rename removes that name first, so for a moment
  neither file is there. A rename inside one directory is always atomic.
- A file that has more than one name (hard links) cannot move to another range: the rename fails
  with `EXDEV`, and `mv` then copies the file. A hard link between two ranges also fails with
  `EXDEV`.
- Directory moves to another parent wait for each other: one client moves a directory at a time in
  the whole namespace. A client that stops while it holds this lock loses it when its session ends
  (30 s).
- A mount that stops in the middle of a change across two ranges (`mkdir`, `rmdir`, `rename`,
  `unlink`) leaves no error. The next lookup of that name finishes or undoes the change.

## Check the range

```sh
giz storage usage
```

`usage` gives the retained chunks, their bytes (each chunk once), the copy count and the stored
bytes (bytes × copies).

The leader checks the copies of each retained chunk every 30 seconds
(`GIZ_STORAGE_SCRUB_INTERVAL_MS`), at most at `GIZ_STORAGE_SCRUB_BYTES_PER_SECOND`. It repairs a
missing or damaged copy from a good copy.

## Limits today

- `copies` and `domains` are fixed at the first start.
- All nodes of a range hold all metadata of that range. More metadata needs more ranges
  ([Add a range to the namespace](#add-a-range-to-the-namespace)). A namespace has at most 31 ranges
  after band 0. A range does not split or move its directories to another range by itself, and the
  entries of one directory are always in one range.
- A six-field descriptor uses TCP with signed requests but no encryption: use it on a private network. A nine-field
  descriptor (`ID,HOST,PORT,PROVIDER,SITE,INCARNATION,tls,SERVER-NAME,CA`, with the CA certificate in base64) and
  `GIZ_STORAGE_TLS_CHAIN` and `GIZ_STORAGE_TLS_KEY` on each node give TLS between nodes and clients.
- A backup and a change of the nodes wait for the tier files to go to chunks. The range asks the
  mount to do it, and the command retries.

## Troubleshooting

| Message | Cause and action |
| --- | --- |
| `storage listener unavailable` | Another program has the port. Choose a port below the ephemeral range, and start again. |
| `storage configuration differs from durable admission` | The node started with other arguments than its first start. Use the first-start arguments. |
| `new learner lacks verified retained byte transfer` | The new node has not received the range's state yet. `topology` retries for 60 seconds. Make sure that the new node runs with `--join`, the same key and history, and a reachable address. |
| `different topology transition is already committed` | Another change runs. Wait for it, or cancel it with `giz storage topology cancel`. |
| `topology lacks minimum byte copies` | The new list has fewer nodes than `copies`. |
| `the range on NODES did not take band N: -17` | That range already has another band or already has files. Use a new, empty range. |
| `filesystem range of band N serves band M` | The root of band 0 lists the wrong nodes for band N. Run `range add N` again with the right node list. |

## How this page is tested

Tests with real processes run every command on this page except `range add`: a node joins (`node start --join`),
`topology` and `topology cancel` move chunks, two `backup` runs and a restore, the disks of a node, and two ranges
(placement, moves of files and directories between them, half-done changes and the lock of directory moves). After a
join, the new node holds chunks and a client of both nodes reads the data.