GizStackGizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/models/ardy/model

NVIDIA's ARDY (ARDY-SOMA-RP-30FPS-Horizon60), a person's motion from a text, made two seconds at a time after the motion so far, as nv-tlabs/ardy computes it. A token is 4 frames: the global root of each (its place x, y, z and heading cos, sin) and 128 latent values of the body. A window denoises 15 new tokens after at most 60 tokens of history in 10 DDIM steps, each step two transformers of 8 layers: the root stage gives the roots, the body stage the latents, reading the roots as local velocities. Classifier-free guidance weighs the text against no text. A causal transformer (the FSQ tokenizer's decoder) gives each frame's body: the joints' places, global rotations (6D), velocities and foot contacts of SOMA's 30 joints. Many people step in one batch.

gizlang
import * as Model from "gizai/models/ardy/model";

Types

Ardy

gizlang
type Ardy = { gpu: Gpu; root: Stage; body: Stage; rootHistory: Affine; rootNew: Affine; bodyHistory: Affine; bodyNew: Affine; rootGoal: Affine; bodyGoal: Affine; decIn: Affine; decCond: Affine; decLayers: Vec<Layer>; decOut: Affine; decPlaces: Tensor; mean: Vec<f64>; std: Vec<f64>; latentMean: Vec<f64>; latentStd: Vec<f64>; }

ARDY on a GPU: its two stages, the projections of a token into each (history and generation: the generation's first columns; its constraint columns, the observed body and the mask of TOKEN frames, apart with no bias), the decoder, and the statistics: of the features (global root, local root, body: 370) and of the latents after quantization.

  • gpu: M7.Gpu
  • root: M79.Stage
  • body: M79.Stage
  • rootHistory: M79.Affine
  • rootNew: M79.Affine
  • bodyHistory: M79.Affine
  • bodyNew: M79.Affine
  • rootGoal: M79.Affine
  • bodyGoal: M79.Affine
  • decIn: M79.Affine
  • decCond: M79.Affine
  • decLayers: Vec<M79.Layer>
  • decOut: M79.Affine
  • decPlaces: _gizlang_lib_ai__tensor__storage.Tensor<f32> — The decoder's places of its tokens ([HISTORY / TOKEN + HORIZON / TOKEN, DEC_WIDTH]).
  • mean: Vec<f64>
  • std: Vec<f64>
  • latentMean: Vec<f64>
  • latentStd: Vec<f64>

Goal

gizlang
type Goal = { frame: i64; x: f64; z: f64; heading?: f64; }

Where a person's root is to be at frame frame of its next window (0 to HORIZON - 1): its place x and z in the world, and its heading (radians from +z toward +x) when given.

  • frame: i64
  • x: f64
  • z: f64
  • heading: _gizlang_sdk_core.Option<f64>

People

gizlang
type People = { tokens: Vec<Vec<f64>>; center: Vec<Vec<f64>>; heading: Vec<f64>; text: Vec<Vec<f32>>; goals: Vec<Vec<Goal>>; }

People in motion, one sequence each: their history of tokens (HYBRID values each, roots about the place center last recentred on: x and z each), their heading at the first token they read, their text vectors ([TEXT]), and the goals of their next window (Ardy.step reads and clears them).

  • tokens: Vec<Vec<f64>>
  • center: Vec<Vec<f64>>
  • heading: Vec<f64>
  • text: Vec<Vec<f32>>
  • goals: Vec<Vec<M79.Goal>>

Functions

load

gizlang
function load(folder: string, gpu: Gpu): Ardy throws string

ARDY of folder (denoiser.safetensors, tokenizer.safetensors, stats/) on gpu.

Effects: fail, read, write, console, io, alloc.

people

gizlang
function people(texts: Vec<Vec<f32>>): People

People with no motion yet, facing +z at the origin, each with its text vector.

Effects: fail, alloc.

frames

gizlang
function frames(features: Vec<f64>): Frames

FEATURES (Ardy.features) as frames of SOMA30 (gizlang/retarget): each joint's turn from its global 6D rotation (the first two columns of its matrix) made local to its parent's, and the root's place.

Effects: fail, read, write, console, io, alloc.

Methods

Ardy.step

gizlang
function Ardy.step(self: Ardy, p: mut People, seed: u64, noise: Option<(person: i64) => Vec<f32>> = undefined): void

The next window of every person: HORIZON new frames as 15 tokens added to its history, its roots recentred on its last frame. noise gives a person's starting noise (15 x HYBRID); none: drawn from seed. A person's goals guide its window as a third pass: no text, the goals observed (classifier-free guidance of the goals, as of the text).

Effects: fail, read, write, console, io, alloc.

Ardy.features

gizlang
function Ardy.features(self: Ardy, p: People, since: i64 = 0): Vec<Vec<f64>>

Every person's frames from frame since (a window's first; 0: all) as features ([frames x REP], not normalized), its roots in the world: each window of 15 tokens decoded after at most HISTORY frames before it, all people in one batch. A game takes each window's new frames with since at its first.

Effects: fail, read, write, console, io, alloc.

Constants

FPS

gizlang
const FPS: f64 = 30.0

The frames a second.

HORIZON

gizlang
const HORIZON: i64 = 60

The new frames of a window (Ardy.step).

JOINTS

gizlang
const JOINTS: Vec<string> = ["Hips", "Spine1", "Spine2", "Chest", "Neck1", "Neck2", "Head", "Jaw", "LeftEye", "RightEye", "LeftShoulder", "LeftArm", "LeftForeArm", "LeftHand", "LeftHandThumbEnd", "LeftHandMiddleEnd", "RightShoulder", "RightArm", "RightForeArm", "RightHand", "RightHandThumbEnd", "RightHandMiddleEnd", "LeftUpperLeg", "LeftLowerLeg", "LeftFoot", "LeftToeBase", "RightUpperLeg", "RightLowerLeg", "RightFoot", "RightToeBase"]

SOMA's 30 joints (ARDY's somaskel30), parents first, and each one's parent. Its thighs and shins are named as VRM names them (SOMA's own names are Leg and Shin).

REP

gizlang
const REP: i64 = 366

A frame's features: root place (3), heading (2), the other joints' places (29 x 3), global rotations (30 x 6), velocities (30 x 3), foot contacts (4); the root's part; the body's part; the local root (rotation velocity, x and z velocity, height).

TEXT

gizlang
const TEXT: i64 = 4096

A text vector's width (LLM2Vec's).