gizai/dist
processes that train as one. A group is size processes; rank 0 is its root, and every other rank holds one TCP connection to it (the SDK's streams, on one host or across hosts) and one to each neighbor of a ring. Data parallel training sums each step's gradients over the group around the ring, at each tensor's owner (Group.scatter), takes the owners' moves (Group.gather), and starts every rank from the root's weights (Group.broadcast); the root gathers the ranks' optimizer state to write it (Group.collect).
A rank waits for its root to listen, up to the group's deadline, so the ranks start in any order and meet again after a loss. Each read and write waits at most the deadline; a peer that does not answer in time, or closes, fails the call with Lost, and the group's run meets again from its folder (gizai/llm/learn's run). A group may bound the bytes a second each rank sends (rate), for a network that others use.
import * as Dist from "gizai/dist";Types
Lost
type Lost = { rank: i64; message: string; }A peer that closed or did not answer within the deadline.
rank:i64message:string
Group
type GroupOne process of a group: its rank of size. The root holds a stream to each other rank, in rank order; another rank
holds one stream, to the root. _next and _prev are the streams to the next rank and from the one before, around
the ring (none alone).
rank:i64size:i64rate:i64
Functions
lost
function lost(error: string): boolWhether error is a Lost.text.
Effects: fail.
alone
function alone(): GroupThe group of one process: Group.sum and Group.broadcast give back what they are given.
join
function join(rank: i64, size: i64, host: string, port: i64, deadline: i64, rate: i64 = 0): Group throws LostRank rank of a group of size whose root listens at host:port. The root accepts the other ranks, each of which
says its rank and the port of its ring listener first; a rank connects to the root. Then the root sends each rank
the ring's addresses, and each connects to the next (the root by host). Every stream waits at most deadline
milliseconds a call. Each rank sends at most rate bytes a second (0: no bound).
Effects: fail, read, io, alloc.
accepted
function accepted(listener: Listener, size: i64, deadline: i64, rate: i64 = 0): Group throws LostThe root of a group of size on a listener that is open already: it accepts the other ranks.
Effects: fail, read, io, alloc.
owners
function owners(parts: Vec<Tensor>, size: i64): Vec<i64>The owners of parts by their sizes: each part to the rank that holds the fewest numbers so far, the largest
first, the same on every rank. Each rank owns at least one part when there are as many parts as ranks.
Effects: fail, alloc.
Methods
Lost.text
function Lost.text(self: Lost): stringThe error text of a lost peer; a run that meets again knows it by lost.
Effects: fail, alloc.
Group.sum
function Group.sum(self: Group, parts: Vec<Tensor>, wire: i64 = TensorCore.BF16): Vec<Tensor> throws LostThe sum over the group of each rank's parts (the same shapes on every rank), on every rank, where each rank's
part is. The parts go as one flat tensor in size segments around the ring: each rank adds the segment the one
before sends it and passes it on, until each segment's sum is whole at one rank (its owner), which then goes round
to every rank (Group.scatter, then Group.gather). A link carries 2 (size - 1) / size of the parts a step, whatever
the size; the root of a star carried 2 (size - 1) times them. A segment from a GPU crosses as wire (BF16 by
default: half the bytes of f32, as a gradient needs), from the host as f32; the adds are f32 where the parts are,
and an owner rounds its sum as it sends it, so every rank ends with the same values.
Effects: fail, read, write, console, io, alloc.
Group.scatter
function Group.scatter(self: Group, parts: Vec<Tensor>, owners: Vec<i64>, wire: i64 = TensorCore.BF16): Vec<Tensor> throws LostThe sums over the group of the parts this rank owns (owners[i] is part i's rank), in their order: each rank
sends (size - 1) / size of the parts, half of Group.sum.
Effects: fail, read, write, console, io, alloc.
Group.gather
function Group.gather(self: Group, parts: Vec<Tensor>, owners: Vec<i64>, wire: i64 = TensorCore.BF16): Vec<Tensor> throws LostEvery part from its owner, on every rank: this rank's own parts are its values, and the others' entries give their
shapes only. An owner's parts cross as wire and it takes them back rounded so, as every rank does.
Effects: fail, read, write, console, io, alloc.
Group.collect
function Group.collect(self: Group, data: bytes): Vec<bytes> throws LostEach rank's data at the root, in rank order (another rank gets none back).
Effects: fail, read, io, alloc.
Group.broadcast
function Group.broadcast(self: Group, data: bytes): bytes throws LostThe root's data on every rank (another rank's data is not read).
Effects: fail, read, io, alloc.