gizai/models/ardy/model
NVIDIA's ARDY (ARDY-SOMA-RP-30FPS-Horizon60), a person's motion from a text, made two seconds at a time after the motion so far, as nv-tlabs/ardy computes it. A token is 4 frames: the global root of each (its place x, y, z and heading cos, sin) and 128 latent values of the body. A window denoises 15 new tokens after at most 60 tokens of history in 10 DDIM steps, each step two transformers of 8 layers: the root stage gives the roots, the body stage the latents, reading the roots as local velocities. Classifier-free guidance weighs the text against no text. A causal transformer (the FSQ tokenizer's decoder) gives each frame's body: the joints' places, global rotations (6D), velocities and foot contacts of SOMA's 30 joints. Many people step in one batch.
import * as Model from "gizai/models/ardy/model";Types
Ardy
type Ardy = { gpu: Gpu; root: Stage; body: Stage; rootHistory: Affine; rootNew: Affine; bodyHistory: Affine; bodyNew: Affine; rootGoal: Affine; bodyGoal: Affine; decIn: Affine; decCond: Affine; decLayers: Vec<Layer>; decOut: Affine; decPlaces: Tensor; mean: Vec<f64>; std: Vec<f64>; latentMean: Vec<f64>; latentStd: Vec<f64>; }ARDY on a GPU: its two stages, the projections of a token into each (history and generation: the generation's first columns; its constraint columns, the observed body and the mask of TOKEN frames, apart with no bias), the decoder, and the statistics: of the features (global root, local root, body: 370) and of the latents after quantization.
gpu:M7.Gpuroot:M79.Stagebody:M79.StagerootHistory:M79.AffinerootNew:M79.AffinebodyHistory:M79.AffinebodyNew:M79.AffinerootGoal:M79.AffinebodyGoal:M79.AffinedecIn:M79.AffinedecCond:M79.AffinedecLayers:Vec<M79.Layer>decOut:M79.AffinedecPlaces:_gizlang_lib_ai__tensor__storage.Tensor<f32>— The decoder's places of its tokens ([HISTORY / TOKEN + HORIZON / TOKEN, DEC_WIDTH]).mean:Vec<f64>std:Vec<f64>latentMean:Vec<f64>latentStd:Vec<f64>
Goal
type Goal = { frame: i64; x: f64; z: f64; heading?: f64; }Where a person's root is to be at frame frame of its next window (0 to HORIZON - 1): its place x and z in the
world, and its heading (radians from +z toward +x) when given.
frame:i64x:f64z:f64heading:_gizlang_sdk_core.Option<f64>
People
type People = { tokens: Vec<Vec<f64>>; center: Vec<Vec<f64>>; heading: Vec<f64>; text: Vec<Vec<f32>>; goals: Vec<Vec<Goal>>; }People in motion, one sequence each: their history of tokens (HYBRID values each, roots about the place center
last recentred on: x and z each), their heading at the first token they read, their text vectors ([TEXT]), and the
goals of their next window (Ardy.step reads and clears them).
tokens:Vec<Vec<f64>>center:Vec<Vec<f64>>heading:Vec<f64>text:Vec<Vec<f32>>goals:Vec<Vec<M79.Goal>>
Functions
load
function load(folder: string, gpu: Gpu): Ardy throws stringARDY of folder (denoiser.safetensors, tokenizer.safetensors, stats/) on gpu.
Effects: fail, read, write, console, io, alloc.
people
function people(texts: Vec<Vec<f32>>): PeoplePeople with no motion yet, facing +z at the origin, each with its text vector.
Effects: fail, alloc.
frames
function frames(features: Vec<f64>): FramesFEATURES (Ardy.features) as frames of SOMA30 (gizlang/retarget): each joint's turn from its global 6D rotation (the first two columns of its matrix) made local to its parent's, and the root's place.
Effects: fail, read, write, console, io, alloc.
Methods
Ardy.step
function Ardy.step(self: Ardy, p: mut People, seed: u64, noise: Option<(person: i64) => Vec<f32>> = undefined): voidThe next window of every person: HORIZON new frames as 15 tokens added to its history, its roots recentred on its
last frame. noise gives a person's starting noise (15 x HYBRID); none: drawn from seed. A person's goals guide
its window as a third pass: no text, the goals observed (classifier-free guidance of the goals, as of the text).
Effects: fail, read, write, console, io, alloc.
Ardy.features
function Ardy.features(self: Ardy, p: People, since: i64 = 0): Vec<Vec<f64>>Every person's frames from frame since (a window's first; 0: all) as features ([frames x REP], not normalized),
its roots in the world: each window of 15 tokens decoded after at most HISTORY frames before it, all people in one
batch. A game takes each window's new frames with since at its first.
Effects: fail, read, write, console, io, alloc.
Constants
FPS
const FPS: f64 = 30.0The frames a second.
HORIZON
const HORIZON: i64 = 60The new frames of a window (Ardy.step).
JOINTS
const JOINTS: Vec<string> = ["Hips", "Spine1", "Spine2", "Chest", "Neck1", "Neck2", "Head", "Jaw", "LeftEye", "RightEye", "LeftShoulder", "LeftArm", "LeftForeArm", "LeftHand", "LeftHandThumbEnd", "LeftHandMiddleEnd", "RightShoulder", "RightArm", "RightForeArm", "RightHand", "RightHandThumbEnd", "RightHandMiddleEnd", "LeftUpperLeg", "LeftLowerLeg", "LeftFoot", "LeftToeBase", "RightUpperLeg", "RightLowerLeg", "RightFoot", "RightToeBase"]SOMA's 30 joints (ARDY's somaskel30), parents first, and each one's parent. Its thighs and shins are named as VRM names them (SOMA's own names are Leg and Shin).
REP
const REP: i64 = 366A frame's features: root place (3), heading (2), the other joints' places (29 x 3), global rotations (30 x 6), velocities (30 x 3), foot contacts (4); the root's part; the body's part; the local root (rotation velocity, x and z velocity, height).
TEXT
const TEXT: i64 = 4096A text vector's width (LLM2Vec's).