# gizai/models/resplat/model

ReSplat (Xu, Barath, Geiger and Pollefeys, "ReSplat: Learning Recurrent Gaussian Splatting", ECCV 2026; code and weights MIT): 3D Gaussians from a few posed pictures in one pass, then refined by their own rendering error. The checkpoint is read as published (resplat-*.pth).

A pass: each view's depth from multi-view stereo (UniMatch's CNN and transformer, a plane-sweep cost volume over log depth, DINOv2 ViT-B's features, a DPT upsampler); a Gaussian a 4 x 4 block of pixels from those features and the depth, through a point transformer over each Gaussian's nearest neighbors; then rounds of: render every view, read the error through ResNet-18's first layers, and add the update a second point transformer gives.

```gizlang
import * as Model from "gizai/models/resplat/model";
```

## Types

### `Model`

```gizlang
type Model = { w: Scope; vit: Vit; gpu: Gpu; }
```

The network on a GPU: its tensors by their checkpoint names less "encoder.".

- `w`: `_gizlang_lib_ai__netops__linear.Scope`
- `vit`: `_gizlang_lib_ai__parts__dinov2.Vit`
- `gpu`: `M7.Gpu`

### `View`

```gizlang
type View = { c2w: Vec<f64>; k: Vec<f64>; near: f64; far: f64; }
```

A posed picture: its camera to world matrix (OpenCV axes, row by row), its intrinsics over the picture's width and
height (row by row), and the depths its scene lies between.

- `c2w`: `Vec<f64>`
- `k`: `Vec<f64>`
- `near`: `f64`
- `far`: `f64`

### `Latent`

```gizlang
type Latent = { features: Tensor; depth: Tensor; condition: Tensor; }
```

One view's maps the Gaussians are made from (all at 1/LATENT), as the depth model leaves them.

- `features`: `_gizlang_lib_ai__tensor__storage.Tensor<f32>`
- `depth`: `_gizlang_lib_ai__tensor__storage.Tensor<f32>`
- `condition`: `_gizlang_lib_ai__tensor__storage.Tensor<f32>`

### `Seen`

```gizlang
type Seen = { picture: Tensor; cnn: Vec<Tensor>; mono: Vec<Tensor>; metric: Tensor; }
```

What ReSplat reads from one picture alone: the picture ([3, H, W], colors 0 to 1; H and W multiples of 32), its CNN
features (1/2, 1/4, 1/8), its DINOv2 taps and its ResNet features at the latent size (the refinement's truth). Views
shared by several runs of a world are read once.

- `picture`: `_gizlang_lib_ai__tensor__storage.Tensor<f32>`
- `cnn`: `Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>`
- `mono`: `Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>`
- `metric`: `_gizlang_lib_ai__tensor__storage.Tensor<f32>`

### `World`

```gizlang
type World = { parts: Vec<Tensor>; views: Vec<Vec<View>>; }
```

A world: Gaussians made ready to draw (splatsOf) for runs of neighboring views, each with its views. A run is right
near its own views only (its depths do not agree with a far run's), so a picture is drawn from the run nearest its
camera, as ReSplat's sliding window does.

- `parts`: `Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>`
- `views`: `Vec<Vec<M56.View>>`

## Functions

### `centerDistance`

```gizlang
function centerDistance(a: Vec<f64>, b: Vec<f64>): f64
```

The distance between the centers of two cameras (camera to world matrices, by rows).

Effects: fail, alloc.

### `spacing`

```gizlang
function spacing(views: Vec<View>): f64
```

The mean distance between neighboring views of a path (0 for one view).

Effects: fail, alloc.

### `inverse`

```gizlang
function inverse(c2w: Vec<f64>): Vec<f64>
```

The inverse of a camera to world matrix (a rigid motion, 16 values by rows): world to camera, R^T and -R^T t.

Effects: fail, alloc.

### `load`

```gizlang
function load(file: string, gpu: Gpu): Model throws string
```

The model of the checkpoint `file` (a ReSplat .pth, or safetensors of the same names) on `gpu`.

Effects: fail, read, write, console, io, alloc.

### `neighbors`

```gizlang
function neighbors(views: Vec<View>, m: i64): Vec<Vec<i64>>
```

The `m` views nearest each view (by camera center, the view itself not among them), nearest first, or all the
others for three views or fewer.

Effects: fail, read, write, alloc.

### `points`

```gizlang
function points(latents: Vec<Latent>, views: Vec<View>): Tensor
```

The world point of each latent pixel of each view ([v h w, 3]): its ray through the pixel's center times its depth.

Effects: fail, read, write, console, io, alloc.

### `quaternion`

```gizlang
function quaternion(r: Vec<f64>): Vec<f64>
```

The quaternion (w, x, y, z) of a rotation matrix (by rows).

Effects: fail, alloc.

### `initial`

```gizlang
function initial(raw: Tensor, depths: Tensor, colors: Tensor, views: Vec<View>, h: i64, w: i64): Tensor
```

The first Gaussians' parameters from the Gaussian head's ([v h w, 58]: rotation, scales, opacity, a center offset
in the pixel, colors), the latent depths ([v h w, 1]) and the picture's mean color over each latent pixel ([v h w,
3]): the center on the ray through the offset pixel center at that depth, scales softplus(s - 4) held to [1e-6, 1],
the opacity held where its sigmoid is 1e-6 from 0 and 1, the first color coefficients plus the mean color's.

Effects: fail, read, write, console, io, alloc.

### `firstOf`

```gizlang
function firstOf(raw: Tensor, latents: Vec<Latent>, pictures: Vec<Tensor>, views: Vec<View>): Tensor
```

initial() of the Gaussian head's output for the views' latents and pictures ([3, H, W] each).

Effects: fail, read, write, console, io, alloc.

### `splatsOf`

```gizlang
function splatsOf(params: Tensor, views: Vec<View>, h: i64, w: i64): Tensor
```

Gaussians for gizai/render/gaussians ([n, splatWidth(DEGREE)]) from their parameters (h w Gaussians a view): log
scales, the rotation turned into the world (w, x, y, z), the colors in a PLY file's order.

Effects: fail, read, write, console, io, alloc.

### `cameraOf`

```gizlang
function cameraOf(view: View, width: i64, height: i64): Tensor
```

The camera tensor of `view` for gizai/render/gaussians, for a width x height picture.

Effects: fail, alloc.

### `drawing`

```gizlang
function drawing(splats: Tensor, view: View, width: i64, height: i64): Tensor
```

The picture of `splats` (from splatsOf) seen from `view` as ReSplat draws it: [height width, 3] rows of colors. A
Gaussian covers tiles in proportion to the picture's area, so the list holds 64 entries a Gaussian at 448 x 256, up to
the renderer's limit.

Effects: fail, read, write, console, io, alloc.

### `entriesOf`

```gizlang
function entriesOf(splats: Tensor, width: i64, height: i64): i64
```

The (Gaussian, tile) entries drawing lists at most for `splats` in a width x height picture.

Effects: fail.

### `rasterOf`

```gizlang
function rasterOf(view: View): Raster
```

How ReSplat draws Gaussians seen from `view`.

### `viewOf`

```gizlang
function viewOf(camera: Vec<f32>, view: View): View
```

`view` with the camera of `camera` ([1, CAMERA] values, as cameraOf makes them; the same k).

Effects: fail, alloc.

### `closed`

```gizlang
function closed(views: Vec<View>): bool
```

Whether a path of views comes back to its start: its last view as near its first as two neighbors are (twice their
mean spacing) and facing the same way within 45 degrees.

Effects: fail, alloc.

### `worldOf`

```gizlang
function worldOf(data: bytes, gpu: Gpu): World throws string
```

The world of a PLY file World.ply wrote, its splats on `gpu`.

Effects: fail, read, write, console, io, alloc.

## Methods

### `Model.normalized`

```gizlang
function Model.normalized(self: Model, images: Tensor): Vec<Tensor>
```

The views ([v, 3, h, w], 0 to 1) less ImageNet's mean, over its spread, one [3, h, w] a view.

Effects: fail, read, write, console, io, alloc.

### `Model.cnn`

```gizlang
function Model.cnn(self: Model, x: Tensor): Vec<Tensor>
```

UniMatch's CNN features of a normalized view ([3, h, w]): 64 channels at 1/2, 96 at 1/4 and 128 at 1/8.

Effects: fail, read, write, console, io, alloc.

### `Model.mono`

```gizlang
function Model.mono(self: Model, views: Vec<Tensor>): Vec<Vec<Tensor>>
```

DINOv2's maps of the normalized views ([768, h / 14, w / 14] after blocks 3, 6, 9 and 12, each view's): the views
resized to multiples of 14 (bilinear, aligned corners) first.

Effects: fail, read, write, console, io, alloc.

### `Model.multiView`

```gizlang
function Model.multiView(self: Model, features: Vec<Tensor>, near: Vec<Vec<i64>>): Vec<Tensor>
```

UniMatch's multi-view transformer over the views' 1/8 features ([128, h, w] each): a sine position within each of
2 x 2 windows, then layers of self attention and of cross attention to the `near` views with an MLP, in k x k windows
(k 2, 4 past 96 columns, 8 past 192), every second layer shifted.

Effects: fail, read, write, console, io, alloc.

### `Model.costs`

```gizlang
function Model.costs(self: Model, features: Vec<Tensor>, views: Vec<View>, near: Vec<Vec<i64>>): Vec<Tensor>
```

The plane-sweep cost of each view against its `near` views from their features ([c, h, w] each, at 1/8): for each
pixel and log-depth candidate, the mean over the neighbors of the dot product (over the square root of c) of the
view's feature with the neighbor's, sampled (bilinear, zero outside) where that point lands in it. [D, h, w] a view.
A bilinear sample's dot product is the same mix of the dot products with the four pixels around it, and the depth
candidates of a pixel land on the same few pixels again and again: so each view's pixels are multiplied with all
its neighbors' pixels once (one product on the matrix cores) and each candidate mixes four of those numbers.

Effects: fail, read, write, console, io, alloc.

### `Model.coarse`

```gizlang
function Model.coarse(self: Model, costs: Vec<Tensor>, cnn: Vec<Tensor>, mv: Vec<Tensor>, mono: Vec<Tensor>, views: Vec<View>): Vec<Vec<Tensor>>
```

The depth regression at 1/8 from the views' costs ([D, h, w]), CNN features, transformer features and the last
DINOv2 map (resized to h x w): the regressor (a convolution, group norm and GELU, the UNet, a convolution, plus a
1 x 1 residual), then the depth head (two 3 x 3 convolutions over repeated edges, GELU between): each view's
probabilities over the candidates ([D, h, w]) and its log depth ([1, h, w], their expectation).

Effects: fail, read, write, console, io, alloc.

### `Model.depths`

```gizlang
function Model.depths(self: Model, coarse: Vec<Vec<Tensor>>, mono: Vec<Vec<Tensor>>, cnn: Vec<Vec<Tensor>>, mv: Vec<Tensor>, views: Vec<View>): Vec<Tensor>
```

Each view's depth at full size ([1, H, W], in scene units): the log depth at 1/8 resized (bilinear, aligned
corners) plus the DPT upsampler's residual, held between log near and log far, then exp. The upsampler reassembles
the four DINOv2 maps (resized to 1/8), joins each with the CNN features of its size (and the 1/8 one with the
transformer features and the log depth) through a 1 x 1 convolution, fuses them coarse to fine and ends in two 3 x 3
convolutions over repeated edges (GELU after each) and a 1 x 1.

Effects: fail, read, write, console, io, alloc.

### `Model.latent`

```gizlang
function Model.latent(self: Model, image: Tensor, depth: Tensor, probs: Tensor, raw: Vec<Tensor>, cnn: Vec<Tensor>, mv: Tensor): Latent
```

View i's inputs to the Gaussian regressor at 1/4: the picture's 4 x 4 blocks as channels, the depth, the most
likely candidate's probability, the DINOv2 maps (each at 1/16, its channels spread over 4 x 4 pixels), the CNN
features and the transformer features; then the regressor (two 3 x 3 convolutions, GELU between) and the condition
the refinement starts from: [regressor out, picture blocks, features, probability].

Effects: fail, read, write, console, io, alloc.

### `Model.pointTransformer`

```gizlang
function Model.pointTransformer(self: Model, x: Tensor, points: Tensor, views: Vec<View>, h: i64, w: i64, k: i64, at: string): Tensor
```

The point transformer `at` (blocks of neighbor attention, each followed by a multi-view block where it has one) over the features
x ([v h w, c]) of the points ([v h w, 3]) of the views' h x w latent grids, with `k` neighbors a point: its output.

Effects: fail, read, write, console, io, alloc.

### `Model.gaussians`

```gizlang
function Model.gaussians(self: Model, latents: Vec<Latent>, views: Vec<View>): Vec<Tensor>
```

The first Gaussians' raw parameters ([v h w, 58]), the features the refinement starts from ([v h w, 512]) and the
points: each latent pixel's condition through proj, plus the point transformer of that, through the Gaussian head
(two linears, GELU between).

Effects: fail, read, write, console, io, alloc.

### `Model.drawn`

```gizlang
function Model.drawn(self: Model, params: Tensor, views: Vec<View>, h: i64, w: i64, view: View, width: i64, height: i64): Tensor
```

The picture ([3, height, width]) of the Gaussians `params` seen from `view`, as ReSplat draws it (black behind).

Effects: fail, read, write, console, io, alloc.

### `Model.refined`

```gizlang
function Model.refined(self: Model, params: Tensor, state0: Tensor, seen: Vec<Seen>, views: Vec<View>, steps: i64): Vec<Tensor>
```

`steps` rounds of refinement of the Gaussians `params` from the state the first pass left ([v h w, 512]), against
the views read alone (`seen`): each round draws every view, reads its error (ResNet features and the color
error of each latent pixel's block, through one multi-view block), runs the update's point transformer over the
Gaussians with their state and error, and adds its head's change. The Gaussians after each round.

Effects: fail, read, write, console, io, alloc.

### `Model.seen`

```gizlang
function Model.seen(self: Model, pictures: Tensor): Vec<Seen>
```

Each picture of `pictures` ([v, 3, H, W]) read alone.

Effects: fail, read, write, console, io, alloc.

### `Model.reconstruct`

```gizlang
function Model.reconstruct(self: Model, seen: Vec<Seen>, views: Vec<View>, steps: i64): Tensor
```

The Gaussians ReSplat makes of posed views read alone (`seen`) after `steps` rounds of refinement: their parameters
([v H W / 16, PARAMS]; splatsOf(..., H / LATENT, W / LATENT) draws them).

Effects: fail, read, write, console, io, alloc.

### `Model.world`

```gizlang
function Model.world(self: Model, pictures: Tensor, views: Vec<View>, steps: i64, group: i64, stride: i64): World
```

The world of posed pictures ([v, 3, H, W]; views in one frame, in their order along the path): ReSplat over runs of
`group` views that start every `stride` views. On an open path the last run ends at the last view; on a closed one
(closed()) the runs go on around the end into the first views, so no seam is left there.

Effects: fail, read, write, console, io, alloc.

### `World.ply`

```gizlang
function World.ply(self: World): bytes
```

The world as a 3DGS PLY file (gizai/render/gaussians' writePly): every run's splats in turn, and in the header a
comment "run N" for each (its splats) followed by a comment "view" for each of its views: c2w (16 values), k (9),
near and far. Any 3DGS viewer shows all runs at once; worldOf reads the runs back.

Effects: fail, read, write, console, io, alloc.

### `World.nearest`

```gizlang
function World.nearest(self: World, view: View): Tensor
```

The run of the world nearest `view`: the least distance from its middle view's center plus how far their forward
axes turn apart (1 - cos, times the runs' mean spacing).

Effects: fail, alloc.

## Constants

### `CANDIDATES`

```gizlang
const CANDIDATES: i64 = 128
```

The depth candidates of the plane sweep, evenly spaced in log depth from near to far.

### `LATENT`

```gizlang
const LATENT: i64 = 4
```

The latent grid's step: a Gaussian a LATENT x LATENT block of pixels.

### `DEGREE`

```gizlang
const DEGREE: i64 = 3
```

The degree of ReSplat's colors.

### `PARAMS`

```gizlang
const PARAMS: i64 = 59
```

ReSplat's Gaussians as it refines them ([n, PARAMS] a Gaussian): the center (world), the scales, the rotation as
an unnormalized quaternion x, y, z, w in its view's camera, the opacity before the sigmoid, and 3 x 16 color
coefficients (r's, g's, b's, each from degree 0 up).

