# gizai/render/gaussians

3D Gaussians drawn to a picture, as 3D Gaussian Splatting (Kerbl et al., 2023) describes: each Gaussian projected to a 2D one, its 16 x 16 pixel tiles listed, the list sorted by tile then depth, and each tile's pixels blended front to back. All in Tile, so the CPU and a GPU give the same picture; a frame reads its camera from a tensor, so a moving camera compiles nothing. A frame's gradient against its Gaussians and camera replays the same steps backward (gizai/render/fit fits Gaussians with it).

```gizlang
import * as Gaussians from "gizai/render/gaussians";
```

## Types

### `Raster`

```gizlang
type Raster = { near: f64; blur: f64; antialiased: bool; }
```

How a Gaussian becomes pixels: Gaussians nearer than `near` are not drawn, `blur` (pixels squared) is added to each
2D covariance, and `antialiased` scales the opacity by the square root of the covariance's determinant before over
after the blur (Mip-Splatting; gsplat's "antialiased"), so a small Gaussian does not grow brighter.

- `near`: `f64`
- `blur`: `f64`
- `antialiased`: `bool`

### `Drawn`

```gizlang
type Drawn
```

A frame drawn (drawn) with what its gradient (Drawn.gradient) reads again: the picture, how much of the background
shows through each pixel ([height width, 1], 1 where no Gaussian covers it), each Gaussian's projected
values, the Gaussians' depth order (places) with each place's first entry and count, the entries' total, each
entry's Gaussian (by its place in the list as made), the entries in the order of their tiles, and each tile's
first, and the chunks of entries each tile took.

- `picture`: `_gizlang_lib_ai__tensor__storage.Tensor<f32>`
- `open`: `_gizlang_lib_ai__tensor__storage.Tensor<f32>`

### `Gradient`

```gizlang
type Gradient = { splats: Tensor; camera: Tensor; }
```

The gradient of a loss of a drawn frame: of its splats ([n, width], as they are) and of its camera ([1, CAMERA]).

- `splats`: `_gizlang_lib_ai__tensor__storage.Tensor<f32>`
- `camera`: `_gizlang_lib_ai__tensor__storage.Tensor<f32>`

## Functions

### `splatWidth`

```gizlang
function splatWidth(d: i64): i64
```

The values of a Gaussian whose colors have spherical harmonics up to degree d.

Effects: fail.

### `classic`

```gizlang
function classic(): Raster
```

3D Gaussian Splatting's own: near 0.2, blur 0.3, no antialiasing.

### `render`

```gizlang
function render(splats: Tensor, camera: Tensor, width: i64, height: i64, background: Vec<f64>, capacity: i64 = 0, raster: Raster = classic()): Tensor
```

The picture ([height width, 3] F32 rows of r, g, b from 0, top row first) of `splats` seen by `camera`, over
`background` (r, g, b). A frame lists at most `capacity` (Gaussian, tile) entries (ENTRIES a Gaussian when 0, below
2^24), nearest Gaussians first: past it, the farthest are not drawn. The Gaussians are sorted by depth, then the
entries by tile alone (stable, so each tile's stay nearest first); only the listed entries are sorted, their
count read on the GPU. Nothing waits for the GPU and every size is fixed by the arguments, so a frame can be
recorded (Gpu.record) and replayed with new splats and camera values.

Effects: fail, read, write, console, io, alloc.

### `drawn`

```gizlang
function drawn(splats: Tensor, camera: Tensor, width: i64, height: i64, background: Vec<f64>, capacity: i64 = 0, raster: Raster = classic()): Drawn
```

render's frame, kept for its gradient.

Effects: fail, read, write, console, io, alloc.

### `pose`

```gizlang
function pose(q: Vec<f64>, pivot: Vec<f64>, move: Vec<f64>): Vec<f32>
```

The pose ([1, POSE] host values) that turns by quaternion `q` about `pivot` and then moves by `move`.

Effects: fail, alloc.

### `posed`

```gizlang
function posed(splats: Tensor, binding: Tensor, poses: Tensor, offsets: Option<Tensor> = undefined): Tensor
```

`splats` posed by linear blend skinning: each splat's center first moved by columns 0 to 2 of its row of
`offsets` ([n, 4], a frame's blendshape motion; none when undefined), then by its `binding` row ([n, 2 BONES]: bone
rows of `poses`, then weights that sum to 1) to the weighted sum of its bones' R x + move, its rotation turned by the
weighted, normalized sum of their quaternions. An object is one bone of weight 1; a body, a rig.

Effects: fail, read, write, console, io, alloc.

### `readPly`

```gizlang
function readPly(data: bytes): Tensor throws string
```

The splats tensor ([n, splatWidth(d)], host) of a 3DGS PLY file (binary, little endian; f_dc colors, and the
f_rest coefficients of degree d when it has them; any other properties skipped).

Effects: fail, alloc.

### `writePly`

```gizlang
function writePly(splats: Tensor, comments: Vec<string>): bytes
```

Splats ([n, splatWidth(d)]) as a 3DGS PLY file (binary, little endian; readPly reads it back): x, y, z, zero
normals, f_dc, f_rest, opacity, scale and rot properties, each a float, with `comments` in its header.

Effects: fail, read, write, console, io, alloc.

### `plyComments`

```gizlang
function plyComments(data: bytes): Vec<string>
```

The comments of a PLY file's header, each without its "comment ".

Effects: fail, alloc.

### `lookAt`

```gizlang
function lookAt(eye: Vec<f64>, target: Vec<f64>, up: Vec<f64>, fov: f64, width: i64, height: i64): Tensor
```

A camera tensor for a camera at `eye` looking at `target` with `up`, a vertical field of view `fov` (radians),
for a width x height picture.

Effects: fail, alloc.

## Methods

### `Drawn.gradient`

```gizlang
function Drawn.gradient(self: Drawn, splats: Tensor, camera: Tensor, dpicture: Tensor, raster: Raster = classic()): Gradient
```

The gradient of a loss against the splats and camera this frame was drawn of (with this raster), from its gradient
against the picture (`dpicture`, the picture's shape). The blend is replayed front to back: an entry's share of the
color behind it is what the picture holds past the colors before it, so nothing is kept a pixel. Each entry's
gradient is summed over its tile's pixels and then over the Gaussian's entries, so the sums take no atomic adds and
do not depend on timing. Left out: a color's change with its direction from the camera.

Effects: fail, read, write, console, io, alloc.

## Constants

### `SPLAT`

```gizlang
const SPLAT: i64 = 14
```

A Gaussian's values in a splats tensor ([n, SPLAT] for colors of degree 0): its center x, y, z; its scales'
logarithms; its rotation (w, x, y, z, any length); its opacity before the sigmoid; its color's first
spherical-harmonic coefficients (r, g, b), as a 3DGS PLY file stores them. Colors of degree d up to 3 add the other
(d + 1)^2 - 1 coefficients of each of r, g and b (a PLY file's f_rest order): splatWidth(d) values.

### `CAMERA`

```gizlang
const CAMERA: i64 = 16
```

A camera tensor ([1, CAMERA]): the world-to-camera rotation (9, by rows) and translation (3), then fx, fy, cx, cy
in pixels. The camera looks along +z with x right and y down.

### `ENTRIES`

```gizlang
const ENTRIES: i64 = 16
```

The (Gaussian, tile) entries a frame lists at most, by default, for each Gaussian.

### `POSE`

```gizlang
const POSE: i64 = 8
```

The values of a pose: a rotation as a quaternion (w, x, y, z), then a move (x, y, z) and a zero: a point x goes
to R x + move.

### `BONES`

```gizlang
const BONES: i64 = 4
```

The bones a splat follows, and the values of its binding: BONES bone rows, then their BONES weights.

