GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/dist

processes that train as one. A group is size processes; rank 0 is its root, and every other rank holds one TCP connection to it (the SDK's streams, on one host or across hosts) and one to each neighbor of a ring. Data parallel training sums each step's gradients over the group around the ring, at each tensor's owner (Group.scatter), takes the owners' moves (Group.gather), and starts every rank from the root's weights (Group.broadcast); the root gathers the ranks' optimizer state to write it (Group.collect).

A rank waits for its root to listen, up to the group's deadline, so the ranks start in any order and meet again after a loss. Each read and write waits at most the deadline; a peer that does not answer in time, or closes, fails the call with Lost, and the group's run meets again from its folder (gizai/llm/learn's run). A group may bound the bytes a second each rank sends (rate), for a network that others use.

gizlang
import * as Dist from "gizai/dist";

Types

Lost

gizlang
type Lost = { rank: i64; message: string; }

A peer that closed or did not answer within the deadline.

  • rank: i64
  • message: string

Group

gizlang
type Group

One process of a group: its rank of size. The root holds a stream to each other rank, in rank order; another rank holds one stream, to the root. _next and _prev are the streams to the next rank and from the one before, around the ring (none alone).

  • rank: i64
  • size: i64
  • rate: i64

Functions

lost

gizlang
function lost(error: string): bool

Whether error is a Lost.text.

Effects: fail.

alone

gizlang
function alone(): Group

The group of one process: Group.sum and Group.broadcast give back what they are given.

join

gizlang
function join(rank: i64, size: i64, host: string, port: i64, deadline: i64, rate: i64 = 0): Group throws Lost

Rank rank of a group of size whose root listens at host:port. The root accepts the other ranks, each of which says its rank and the port of its ring listener first; a rank connects to the root. Then the root sends each rank the ring's addresses, and each connects to the next (the root by host). Every stream waits at most deadline milliseconds a call. Each rank sends at most rate bytes a second (0: no bound).

Effects: fail, read, io, alloc.

accepted

gizlang
function accepted(listener: Listener, size: i64, deadline: i64, rate: i64 = 0): Group throws Lost

The root of a group of size on a listener that is open already: it accepts the other ranks.

Effects: fail, read, io, alloc.

owners

gizlang
function owners(parts: Vec<Tensor>, size: i64): Vec<i64>

The owners of parts by their sizes: each part to the rank that holds the fewest numbers so far, the largest first, the same on every rank. Each rank owns at least one part when there are as many parts as ranks.

Effects: fail, alloc.

Methods

Lost.text

gizlang
function Lost.text(self: Lost): string

The error text of a lost peer; a run that meets again knows it by lost.

Effects: fail, alloc.

Group.sum

gizlang
function Group.sum(self: Group, parts: Vec<Tensor>, wire: i64 = TensorCore.BF16): Vec<Tensor> throws Lost

The sum over the group of each rank's parts (the same shapes on every rank), on every rank, where each rank's part is. The parts go as one flat tensor in size segments around the ring: each rank adds the segment the one before sends it and passes it on, until each segment's sum is whole at one rank (its owner), which then goes round to every rank (Group.scatter, then Group.gather). A link carries 2 (size - 1) / size of the parts a step, whatever the size; the root of a star carried 2 (size - 1) times them. A segment from a GPU crosses as wire (BF16 by default: half the bytes of f32, as a gradient needs), from the host as f32; the adds are f32 where the parts are, and an owner rounds its sum as it sends it, so every rank ends with the same values.

Effects: fail, read, write, console, io, alloc.

Group.scatter

gizlang
function Group.scatter(self: Group, parts: Vec<Tensor>, owners: Vec<i64>, wire: i64 = TensorCore.BF16): Vec<Tensor> throws Lost

The sums over the group of the parts this rank owns (owners[i] is part i's rank), in their order: each rank sends (size - 1) / size of the parts, half of Group.sum.

Effects: fail, read, write, console, io, alloc.

Group.gather

gizlang
function Group.gather(self: Group, parts: Vec<Tensor>, owners: Vec<i64>, wire: i64 = TensorCore.BF16): Vec<Tensor> throws Lost

Every part from its owner, on every rank: this rank's own parts are its values, and the others' entries give their shapes only. An owner's parts cross as wire and it takes them back rounded so, as every rank does.

Effects: fail, read, write, console, io, alloc.

Group.collect

gizlang
function Group.collect(self: Group, data: bytes): Vec<bytes> throws Lost

Each rank's data at the root, in rank order (another rank gets none back).

Effects: fail, read, io, alloc.

Group.broadcast

gizlang
function Group.broadcast(self: Group, data: bytes): bytes throws Lost

The root's data on every rank (another rank's data is not read).

Effects: fail, read, io, alloc.