# gizai/dist

processes that train as one. A group is `size` processes; rank 0 is its root, and every other rank holds one TCP connection to it (the SDK's streams, on one host or across hosts) and one to each neighbor of a ring. Data parallel training sums each step's gradients over the group around the ring, at each tensor's owner (Group.scatter), takes the owners' moves (Group.gather), and starts every rank from the root's weights (Group.broadcast); the root gathers the ranks' optimizer state to write it (Group.collect).

A rank waits for its root to listen, up to the group's deadline, so the ranks start in any order and meet again after a loss. Each read and write waits at most the deadline; a peer that does not answer in time, or closes, fails the call with Lost, and the group's run meets again from its folder (gizai/llm/learn's run). A group may bound the bytes a second each rank sends (`rate`), for a network that others use.

```gizlang
import * as Dist from "gizai/dist";
```

## Types

### `Lost`

```gizlang
type Lost = { rank: i64; message: string; }
```

A peer that closed or did not answer within the deadline.

- `rank`: `i64`
- `message`: `string`

### `Group`

```gizlang
type Group
```

One process of a group: its rank of `size`. The root holds a stream to each other rank, in rank order; another rank
holds one stream, to the root. `_next` and `_prev` are the streams to the next rank and from the one before, around
the ring (none alone).

- `rank`: `i64`
- `size`: `i64`
- `rate`: `i64`

## Functions

### `lost`

```gizlang
function lost(error: string): bool
```

Whether `error` is a Lost.text.

Effects: fail.

### `alone`

```gizlang
function alone(): Group
```

The group of one process: Group.sum and Group.broadcast give back what they are given.

### `join`

```gizlang
function join(rank: i64, size: i64, host: string, port: i64, deadline: i64, rate: i64 = 0): Group throws Lost
```

Rank `rank` of a group of `size` whose root listens at host:port. The root accepts the other ranks, each of which
says its rank and the port of its ring listener first; a rank connects to the root. Then the root sends each rank
the ring's addresses, and each connects to the next (the root by `host`). Every stream waits at most `deadline`
milliseconds a call. Each rank sends at most `rate` bytes a second (0: no bound).

Effects: fail, read, io, alloc.

### `accepted`

```gizlang
function accepted(listener: Listener, size: i64, deadline: i64, rate: i64 = 0): Group throws Lost
```

The root of a group of `size` on a listener that is open already: it accepts the other ranks.

Effects: fail, read, io, alloc.

### `owners`

```gizlang
function owners(parts: Vec<Tensor>, size: i64): Vec<i64>
```

The owners of `parts` by their sizes: each part to the rank that holds the fewest numbers so far, the largest
first, the same on every rank. Each rank owns at least one part when there are as many parts as ranks.

Effects: fail, alloc.

## Methods

### `Lost.text`

```gizlang
function Lost.text(self: Lost): string
```

The error text of a lost peer; a run that meets again knows it by `lost`.

Effects: fail, alloc.

### `Group.sum`

```gizlang
function Group.sum(self: Group, parts: Vec<Tensor>, wire: i64 = TensorCore.BF16): Vec<Tensor> throws Lost
```

The sum over the group of each rank's `parts` (the same shapes on every rank), on every rank, where each rank's
part is. The parts go as one flat tensor in `size` segments around the ring: each rank adds the segment the one
before sends it and passes it on, until each segment's sum is whole at one rank (its owner), which then goes round
to every rank (Group.scatter, then Group.gather). A link carries 2 (size - 1) / size of the parts a step, whatever
the size; the root of a star carried 2 (size - 1) times them. A segment from a GPU crosses as `wire` (BF16 by
default: half the bytes of f32, as a gradient needs), from the host as f32; the adds are f32 where the parts are,
and an owner rounds its sum as it sends it, so every rank ends with the same values.

Effects: fail, read, write, console, io, alloc.

### `Group.scatter`

```gizlang
function Group.scatter(self: Group, parts: Vec<Tensor>, owners: Vec<i64>, wire: i64 = TensorCore.BF16): Vec<Tensor> throws Lost
```

The sums over the group of the parts this rank owns (`owners[i]` is part i's rank), in their order: each rank
sends (size - 1) / size of the parts, half of Group.sum.

Effects: fail, read, write, console, io, alloc.

### `Group.gather`

```gizlang
function Group.gather(self: Group, parts: Vec<Tensor>, owners: Vec<i64>, wire: i64 = TensorCore.BF16): Vec<Tensor> throws Lost
```

Every part from its owner, on every rank: this rank's own parts are its values, and the others' entries give their
shapes only. An owner's parts cross as `wire` and it takes them back rounded so, as every rank does.

Effects: fail, read, write, console, io, alloc.

### `Group.collect`

```gizlang
function Group.collect(self: Group, data: bytes): Vec<bytes> throws Lost
```

Each rank's `data` at the root, in rank order (another rank gets none back).

Effects: fail, read, io, alloc.

### `Group.broadcast`

```gizlang
function Group.broadcast(self: Group, data: bytes): bytes throws Lost
```

The root's `data` on every rank (another rank's `data` is not read).

Effects: fail, read, io, alloc.

