gizai/render/gaussians
3D Gaussians drawn to a picture, as 3D Gaussian Splatting (Kerbl et al., 2023) describes: each Gaussian projected to a 2D one, its 16 x 16 pixel tiles listed, the list sorted by tile then depth, and each tile's pixels blended front to back. All in Tile, so the CPU and a GPU give the same picture; a frame reads its camera from a tensor, so a moving camera compiles nothing. A frame's gradient against its Gaussians and camera replays the same steps backward (gizai/render/fit fits Gaussians with it).
import * as Gaussians from "gizai/render/gaussians";Types
Raster
type Raster = { near: f64; blur: f64; antialiased: bool; }How a Gaussian becomes pixels: Gaussians nearer than near are not drawn, blur (pixels squared) is added to each
2D covariance, and antialiased scales the opacity by the square root of the covariance's determinant before over
after the blur (Mip-Splatting; gsplat's "antialiased"), so a small Gaussian does not grow brighter.
near:f64blur:f64antialiased:bool
Drawn
type DrawnA frame drawn (drawn) with what its gradient (Drawn.gradient) reads again: the picture, how much of the background shows through each pixel ([height width, 1], 1 where no Gaussian covers it), each Gaussian's projected values, the Gaussians' depth order (places) with each place's first entry and count, the entries' total, each entry's Gaussian (by its place in the list as made), the entries in the order of their tiles, and each tile's first, and the chunks of entries each tile took.
picture:_gizlang_lib_ai__tensor__storage.Tensor<f32>open:_gizlang_lib_ai__tensor__storage.Tensor<f32>
Gradient
type Gradient = { splats: Tensor; camera: Tensor; }The gradient of a loss of a drawn frame: of its splats ([n, width], as they are) and of its camera ([1, CAMERA]).
splats:_gizlang_lib_ai__tensor__storage.Tensor<f32>camera:_gizlang_lib_ai__tensor__storage.Tensor<f32>
Functions
splatWidth
function splatWidth(d: i64): i64The values of a Gaussian whose colors have spherical harmonics up to degree d.
Effects: fail.
classic
function classic(): Raster3D Gaussian Splatting's own: near 0.2, blur 0.3, no antialiasing.
render
function render(splats: Tensor, camera: Tensor, width: i64, height: i64, background: Vec<f64>, capacity: i64 = 0, raster: Raster = classic()): TensorThe picture ([height width, 3] F32 rows of r, g, b from 0, top row first) of splats seen by camera, over
background (r, g, b). A frame lists at most capacity (Gaussian, tile) entries (ENTRIES a Gaussian when 0, below
2^24), nearest Gaussians first: past it, the farthest are not drawn. The Gaussians are sorted by depth, then the
entries by tile alone (stable, so each tile's stay nearest first); only the listed entries are sorted, their
count read on the GPU. Nothing waits for the GPU and every size is fixed by the arguments, so a frame can be
recorded (Gpu.record) and replayed with new splats and camera values.
Effects: fail, read, write, console, io, alloc.
drawn
function drawn(splats: Tensor, camera: Tensor, width: i64, height: i64, background: Vec<f64>, capacity: i64 = 0, raster: Raster = classic()): Drawnrender's frame, kept for its gradient.
Effects: fail, read, write, console, io, alloc.
pose
function pose(q: Vec<f64>, pivot: Vec<f64>, move: Vec<f64>): Vec<f32>The pose ([1, POSE] host values) that turns by quaternion q about pivot and then moves by move.
Effects: fail, alloc.
posed
function posed(splats: Tensor, binding: Tensor, poses: Tensor, offsets: Option<Tensor> = undefined): Tensorsplats posed by linear blend skinning: each splat's center first moved by columns 0 to 2 of its row of
offsets ([n, 4], a frame's blendshape motion; none when undefined), then by its binding row ([n, 2 BONES]: bone
rows of poses, then weights that sum to 1) to the weighted sum of its bones' R x + move, its rotation turned by the
weighted, normalized sum of their quaternions. An object is one bone of weight 1; a body, a rig.
Effects: fail, read, write, console, io, alloc.
readPly
function readPly(data: bytes): Tensor throws stringThe splats tensor ([n, splatWidth(d)], host) of a 3DGS PLY file (binary, little endian; f_dc colors, and the f_rest coefficients of degree d when it has them; any other properties skipped).
Effects: fail, alloc.
writePly
function writePly(splats: Tensor, comments: Vec<string>): bytesSplats ([n, splatWidth(d)]) as a 3DGS PLY file (binary, little endian; readPly reads it back): x, y, z, zero
normals, f_dc, f_rest, opacity, scale and rot properties, each a float, with comments in its header.
Effects: fail, read, write, console, io, alloc.
plyComments
function plyComments(data: bytes): Vec<string>The comments of a PLY file's header, each without its "comment ".
Effects: fail, alloc.
lookAt
function lookAt(eye: Vec<f64>, target: Vec<f64>, up: Vec<f64>, fov: f64, width: i64, height: i64): TensorA camera tensor for a camera at eye looking at target with up, a vertical field of view fov (radians),
for a width x height picture.
Effects: fail, alloc.
Methods
Drawn.gradient
function Drawn.gradient(self: Drawn, splats: Tensor, camera: Tensor, dpicture: Tensor, raster: Raster = classic()): GradientThe gradient of a loss against the splats and camera this frame was drawn of (with this raster), from its gradient
against the picture (dpicture, the picture's shape). The blend is replayed front to back: an entry's share of the
color behind it is what the picture holds past the colors before it, so nothing is kept a pixel. Each entry's
gradient is summed over its tile's pixels and then over the Gaussian's entries, so the sums take no atomic adds and
do not depend on timing. Left out: a color's change with its direction from the camera.
Effects: fail, read, write, console, io, alloc.
Constants
SPLAT
const SPLAT: i64 = 14A Gaussian's values in a splats tensor ([n, SPLAT] for colors of degree 0): its center x, y, z; its scales' logarithms; its rotation (w, x, y, z, any length); its opacity before the sigmoid; its color's first spherical-harmonic coefficients (r, g, b), as a 3DGS PLY file stores them. Colors of degree d up to 3 add the other (d + 1)^2 - 1 coefficients of each of r, g and b (a PLY file's f_rest order): splatWidth(d) values.
CAMERA
const CAMERA: i64 = 16A camera tensor ([1, CAMERA]): the world-to-camera rotation (9, by rows) and translation (3), then fx, fy, cx, cy in pixels. The camera looks along +z with x right and y down.
ENTRIES
const ENTRIES: i64 = 16The (Gaussian, tile) entries a frame lists at most, by default, for each Gaussian.
POSE
const POSE: i64 = 8The values of a pose: a rotation as a quaternion (w, x, y, z), then a move (x, y, z) and a zero: a point x goes to R x + move.
BONES
const BONES: i64 = 4The bones a splat follows, and the values of its binding: BONES bone rows, then their BONES weights.