GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/render/gaussians

3D Gaussians drawn to a picture, as 3D Gaussian Splatting (Kerbl et al., 2023) describes: each Gaussian projected to a 2D one, its 16 x 16 pixel tiles listed, the list sorted by tile then depth, and each tile's pixels blended front to back. All in Tile, so the CPU and a GPU give the same picture; a frame reads its camera from a tensor, so a moving camera compiles nothing. A frame's gradient against its Gaussians and camera replays the same steps backward (gizai/render/fit fits Gaussians with it).

gizlang
import * as Gaussians from "gizai/render/gaussians";

Types

Raster

gizlang
type Raster = { near: f64; blur: f64; antialiased: bool; }

How a Gaussian becomes pixels: Gaussians nearer than near are not drawn, blur (pixels squared) is added to each 2D covariance, and antialiased scales the opacity by the square root of the covariance's determinant before over after the blur (Mip-Splatting; gsplat's "antialiased"), so a small Gaussian does not grow brighter.

  • near: f64
  • blur: f64
  • antialiased: bool

Drawn

gizlang
type Drawn

A frame drawn (drawn) with what its gradient (Drawn.gradient) reads again: the picture, how much of the background shows through each pixel ([height width, 1], 1 where no Gaussian covers it), each Gaussian's projected values, the Gaussians' depth order (places) with each place's first entry and count, the entries' total, each entry's Gaussian (by its place in the list as made), the entries in the order of their tiles, and each tile's first, and the chunks of entries each tile took.

  • picture: _gizlang_lib_ai__tensor__storage.Tensor<f32>
  • open: _gizlang_lib_ai__tensor__storage.Tensor<f32>

Gradient

gizlang
type Gradient = { splats: Tensor; camera: Tensor; }

The gradient of a loss of a drawn frame: of its splats ([n, width], as they are) and of its camera ([1, CAMERA]).

  • splats: _gizlang_lib_ai__tensor__storage.Tensor<f32>
  • camera: _gizlang_lib_ai__tensor__storage.Tensor<f32>

Functions

splatWidth

gizlang
function splatWidth(d: i64): i64

The values of a Gaussian whose colors have spherical harmonics up to degree d.

Effects: fail.

classic

gizlang
function classic(): Raster

3D Gaussian Splatting's own: near 0.2, blur 0.3, no antialiasing.

render

gizlang
function render(splats: Tensor, camera: Tensor, width: i64, height: i64, background: Vec<f64>, capacity: i64 = 0, raster: Raster = classic()): Tensor

The picture ([height width, 3] F32 rows of r, g, b from 0, top row first) of splats seen by camera, over background (r, g, b). A frame lists at most capacity (Gaussian, tile) entries (ENTRIES a Gaussian when 0, below 2^24), nearest Gaussians first: past it, the farthest are not drawn. The Gaussians are sorted by depth, then the entries by tile alone (stable, so each tile's stay nearest first); only the listed entries are sorted, their count read on the GPU. Nothing waits for the GPU and every size is fixed by the arguments, so a frame can be recorded (Gpu.record) and replayed with new splats and camera values.

Effects: fail, read, write, console, io, alloc.

drawn

gizlang
function drawn(splats: Tensor, camera: Tensor, width: i64, height: i64, background: Vec<f64>, capacity: i64 = 0, raster: Raster = classic()): Drawn

render's frame, kept for its gradient.

Effects: fail, read, write, console, io, alloc.

pose

gizlang
function pose(q: Vec<f64>, pivot: Vec<f64>, move: Vec<f64>): Vec<f32>

The pose ([1, POSE] host values) that turns by quaternion q about pivot and then moves by move.

Effects: fail, alloc.

posed

gizlang
function posed(splats: Tensor, binding: Tensor, poses: Tensor, offsets: Option<Tensor> = undefined): Tensor

splats posed by linear blend skinning: each splat's center first moved by columns 0 to 2 of its row of offsets ([n, 4], a frame's blendshape motion; none when undefined), then by its binding row ([n, 2 BONES]: bone rows of poses, then weights that sum to 1) to the weighted sum of its bones' R x + move, its rotation turned by the weighted, normalized sum of their quaternions. An object is one bone of weight 1; a body, a rig.

Effects: fail, read, write, console, io, alloc.

readPly

gizlang
function readPly(data: bytes): Tensor throws string

The splats tensor ([n, splatWidth(d)], host) of a 3DGS PLY file (binary, little endian; f_dc colors, and the f_rest coefficients of degree d when it has them; any other properties skipped).

Effects: fail, alloc.

writePly

gizlang
function writePly(splats: Tensor, comments: Vec<string>): bytes

Splats ([n, splatWidth(d)]) as a 3DGS PLY file (binary, little endian; readPly reads it back): x, y, z, zero normals, f_dc, f_rest, opacity, scale and rot properties, each a float, with comments in its header.

Effects: fail, read, write, console, io, alloc.

plyComments

gizlang
function plyComments(data: bytes): Vec<string>

The comments of a PLY file's header, each without its "comment ".

Effects: fail, alloc.

lookAt

gizlang
function lookAt(eye: Vec<f64>, target: Vec<f64>, up: Vec<f64>, fov: f64, width: i64, height: i64): Tensor

A camera tensor for a camera at eye looking at target with up, a vertical field of view fov (radians), for a width x height picture.

Effects: fail, alloc.

Methods

Drawn.gradient

gizlang
function Drawn.gradient(self: Drawn, splats: Tensor, camera: Tensor, dpicture: Tensor, raster: Raster = classic()): Gradient

The gradient of a loss against the splats and camera this frame was drawn of (with this raster), from its gradient against the picture (dpicture, the picture's shape). The blend is replayed front to back: an entry's share of the color behind it is what the picture holds past the colors before it, so nothing is kept a pixel. Each entry's gradient is summed over its tile's pixels and then over the Gaussian's entries, so the sums take no atomic adds and do not depend on timing. Left out: a color's change with its direction from the camera.

Effects: fail, read, write, console, io, alloc.

Constants

SPLAT

gizlang
const SPLAT: i64 = 14

A Gaussian's values in a splats tensor ([n, SPLAT] for colors of degree 0): its center x, y, z; its scales' logarithms; its rotation (w, x, y, z, any length); its opacity before the sigmoid; its color's first spherical-harmonic coefficients (r, g, b), as a 3DGS PLY file stores them. Colors of degree d up to 3 add the other (d + 1)^2 - 1 coefficients of each of r, g and b (a PLY file's f_rest order): splatWidth(d) values.

CAMERA

gizlang
const CAMERA: i64 = 16

A camera tensor ([1, CAMERA]): the world-to-camera rotation (9, by rows) and translation (3), then fx, fy, cx, cy in pixels. The camera looks along +z with x right and y down.

ENTRIES

gizlang
const ENTRIES: i64 = 16

The (Gaussian, tile) entries a frame lists at most, by default, for each Gaussian.

POSE

gizlang
const POSE: i64 = 8

The values of a pose: a rotation as a quaternion (w, x, y, z), then a move (x, y, z) and a zero: a point x goes to R x + move.

BONES

gizlang
const BONES: i64 = 4

The bones a splat follows, and the values of its binding: BONES bone rows, then their BONES weights.