gizai/models/resplat/model
ReSplat (Xu, Barath, Geiger and Pollefeys, "ReSplat: Learning Recurrent Gaussian Splatting", ECCV 2026; code and weights MIT): 3D Gaussians from a few posed pictures in one pass, then refined by their own rendering error. The checkpoint is read as published (resplat-*.pth).
A pass: each view's depth from multi-view stereo (UniMatch's CNN and transformer, a plane-sweep cost volume over log depth, DINOv2 ViT-B's features, a DPT upsampler); a Gaussian a 4 x 4 block of pixels from those features and the depth, through a point transformer over each Gaussian's nearest neighbors; then rounds of: render every view, read the error through ResNet-18's first layers, and add the update a second point transformer gives.
import * as Model from "gizai/models/resplat/model";Types
Model
type Model = { w: Scope; vit: Vit; gpu: Gpu; }The network on a GPU: its tensors by their checkpoint names less "encoder.".
w:_gizlang_lib_ai__netops__linear.Scopevit:_gizlang_lib_ai__parts__dinov2.Vitgpu:M7.Gpu
View
type View = { c2w: Vec<f64>; k: Vec<f64>; near: f64; far: f64; }A posed picture: its camera to world matrix (OpenCV axes, row by row), its intrinsics over the picture's width and height (row by row), and the depths its scene lies between.
c2w:Vec<f64>k:Vec<f64>near:f64far:f64
Latent
type Latent = { features: Tensor; depth: Tensor; condition: Tensor; }One view's maps the Gaussians are made from (all at 1/LATENT), as the depth model leaves them.
features:_gizlang_lib_ai__tensor__storage.Tensor<f32>depth:_gizlang_lib_ai__tensor__storage.Tensor<f32>condition:_gizlang_lib_ai__tensor__storage.Tensor<f32>
Seen
type Seen = { picture: Tensor; cnn: Vec<Tensor>; mono: Vec<Tensor>; metric: Tensor; }What ReSplat reads from one picture alone: the picture ([3, H, W], colors 0 to 1; H and W multiples of 32), its CNN features (1/2, 1/4, 1/8), its DINOv2 taps and its ResNet features at the latent size (the refinement's truth). Views shared by several runs of a world are read once.
picture:_gizlang_lib_ai__tensor__storage.Tensor<f32>cnn:Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>mono:Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>metric:_gizlang_lib_ai__tensor__storage.Tensor<f32>
World
type World = { parts: Vec<Tensor>; views: Vec<Vec<View>>; }A world: Gaussians made ready to draw (splatsOf) for runs of neighboring views, each with its views. A run is right near its own views only (its depths do not agree with a far run's), so a picture is drawn from the run nearest its camera, as ReSplat's sliding window does.
parts:Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>views:Vec<Vec<M56.View>>
Functions
centerDistance
function centerDistance(a: Vec<f64>, b: Vec<f64>): f64The distance between the centers of two cameras (camera to world matrices, by rows).
Effects: fail, alloc.
spacing
function spacing(views: Vec<View>): f64The mean distance between neighboring views of a path (0 for one view).
Effects: fail, alloc.
inverse
function inverse(c2w: Vec<f64>): Vec<f64>The inverse of a camera to world matrix (a rigid motion, 16 values by rows): world to camera, R^T and -R^T t.
Effects: fail, alloc.
load
function load(file: string, gpu: Gpu): Model throws stringThe model of the checkpoint file (a ReSplat .pth, or safetensors of the same names) on gpu.
Effects: fail, read, write, console, io, alloc.
neighbors
function neighbors(views: Vec<View>, m: i64): Vec<Vec<i64>>The m views nearest each view (by camera center, the view itself not among them), nearest first, or all the
others for three views or fewer.
Effects: fail, read, write, alloc.
points
function points(latents: Vec<Latent>, views: Vec<View>): TensorThe world point of each latent pixel of each view ([v h w, 3]): its ray through the pixel's center times its depth.
Effects: fail, read, write, console, io, alloc.
quaternion
function quaternion(r: Vec<f64>): Vec<f64>The quaternion (w, x, y, z) of a rotation matrix (by rows).
Effects: fail, alloc.
initial
function initial(raw: Tensor, depths: Tensor, colors: Tensor, views: Vec<View>, h: i64, w: i64): TensorThe first Gaussians' parameters from the Gaussian head's ([v h w, 58]: rotation, scales, opacity, a center offset in the pixel, colors), the latent depths ([v h w, 1]) and the picture's mean color over each latent pixel ([v h w, 3]): the center on the ray through the offset pixel center at that depth, scales softplus(s - 4) held to [1e-6, 1], the opacity held where its sigmoid is 1e-6 from 0 and 1, the first color coefficients plus the mean color's.
Effects: fail, read, write, console, io, alloc.
firstOf
function firstOf(raw: Tensor, latents: Vec<Latent>, pictures: Vec<Tensor>, views: Vec<View>): Tensorinitial() of the Gaussian head's output for the views' latents and pictures ([3, H, W] each).
Effects: fail, read, write, console, io, alloc.
splatsOf
function splatsOf(params: Tensor, views: Vec<View>, h: i64, w: i64): TensorGaussians for gizai/render/gaussians ([n, splatWidth(DEGREE)]) from their parameters (h w Gaussians a view): log scales, the rotation turned into the world (w, x, y, z), the colors in a PLY file's order.
Effects: fail, read, write, console, io, alloc.
cameraOf
function cameraOf(view: View, width: i64, height: i64): TensorThe camera tensor of view for gizai/render/gaussians, for a width x height picture.
Effects: fail, alloc.
drawing
function drawing(splats: Tensor, view: View, width: i64, height: i64): TensorThe picture of splats (from splatsOf) seen from view as ReSplat draws it: [height width, 3] rows of colors. A
Gaussian covers tiles in proportion to the picture's area, so the list holds 64 entries a Gaussian at 448 x 256, up to
the renderer's limit.
Effects: fail, read, write, console, io, alloc.
entriesOf
function entriesOf(splats: Tensor, width: i64, height: i64): i64The (Gaussian, tile) entries drawing lists at most for splats in a width x height picture.
Effects: fail.
rasterOf
function rasterOf(view: View): RasterHow ReSplat draws Gaussians seen from view.
viewOf
function viewOf(camera: Vec<f32>, view: View): Viewview with the camera of camera ([1, CAMERA] values, as cameraOf makes them; the same k).
Effects: fail, alloc.
closed
function closed(views: Vec<View>): boolWhether a path of views comes back to its start: its last view as near its first as two neighbors are (twice their mean spacing) and facing the same way within 45 degrees.
Effects: fail, alloc.
worldOf
function worldOf(data: bytes, gpu: Gpu): World throws stringThe world of a PLY file World.ply wrote, its splats on gpu.
Effects: fail, read, write, console, io, alloc.
Methods
Model.normalized
function Model.normalized(self: Model, images: Tensor): Vec<Tensor>The views ([v, 3, h, w], 0 to 1) less ImageNet's mean, over its spread, one [3, h, w] a view.
Effects: fail, read, write, console, io, alloc.
Model.cnn
function Model.cnn(self: Model, x: Tensor): Vec<Tensor>UniMatch's CNN features of a normalized view ([3, h, w]): 64 channels at 1/2, 96 at 1/4 and 128 at 1/8.
Effects: fail, read, write, console, io, alloc.
Model.mono
function Model.mono(self: Model, views: Vec<Tensor>): Vec<Vec<Tensor>>DINOv2's maps of the normalized views ([768, h / 14, w / 14] after blocks 3, 6, 9 and 12, each view's): the views resized to multiples of 14 (bilinear, aligned corners) first.
Effects: fail, read, write, console, io, alloc.
Model.multiView
function Model.multiView(self: Model, features: Vec<Tensor>, near: Vec<Vec<i64>>): Vec<Tensor>UniMatch's multi-view transformer over the views' 1/8 features ([128, h, w] each): a sine position within each of
2 x 2 windows, then layers of self attention and of cross attention to the near views with an MLP, in k x k windows
(k 2, 4 past 96 columns, 8 past 192), every second layer shifted.
Effects: fail, read, write, console, io, alloc.
Model.costs
function Model.costs(self: Model, features: Vec<Tensor>, views: Vec<View>, near: Vec<Vec<i64>>): Vec<Tensor>The plane-sweep cost of each view against its near views from their features ([c, h, w] each, at 1/8): for each
pixel and log-depth candidate, the mean over the neighbors of the dot product (over the square root of c) of the
view's feature with the neighbor's, sampled (bilinear, zero outside) where that point lands in it. [D, h, w] a view.
A bilinear sample's dot product is the same mix of the dot products with the four pixels around it, and the depth
candidates of a pixel land on the same few pixels again and again: so each view's pixels are multiplied with all
its neighbors' pixels once (one product on the matrix cores) and each candidate mixes four of those numbers.
Effects: fail, read, write, console, io, alloc.
Model.coarse
function Model.coarse(self: Model, costs: Vec<Tensor>, cnn: Vec<Tensor>, mv: Vec<Tensor>, mono: Vec<Tensor>, views: Vec<View>): Vec<Vec<Tensor>>The depth regression at 1/8 from the views' costs ([D, h, w]), CNN features, transformer features and the last DINOv2 map (resized to h x w): the regressor (a convolution, group norm and GELU, the UNet, a convolution, plus a 1 x 1 residual), then the depth head (two 3 x 3 convolutions over repeated edges, GELU between): each view's probabilities over the candidates ([D, h, w]) and its log depth ([1, h, w], their expectation).
Effects: fail, read, write, console, io, alloc.
Model.depths
function Model.depths(self: Model, coarse: Vec<Vec<Tensor>>, mono: Vec<Vec<Tensor>>, cnn: Vec<Vec<Tensor>>, mv: Vec<Tensor>, views: Vec<View>): Vec<Tensor>Each view's depth at full size ([1, H, W], in scene units): the log depth at 1/8 resized (bilinear, aligned corners) plus the DPT upsampler's residual, held between log near and log far, then exp. The upsampler reassembles the four DINOv2 maps (resized to 1/8), joins each with the CNN features of its size (and the 1/8 one with the transformer features and the log depth) through a 1 x 1 convolution, fuses them coarse to fine and ends in two 3 x 3 convolutions over repeated edges (GELU after each) and a 1 x 1.
Effects: fail, read, write, console, io, alloc.
Model.latent
function Model.latent(self: Model, image: Tensor, depth: Tensor, probs: Tensor, raw: Vec<Tensor>, cnn: Vec<Tensor>, mv: Tensor): LatentView i's inputs to the Gaussian regressor at 1/4: the picture's 4 x 4 blocks as channels, the depth, the most likely candidate's probability, the DINOv2 maps (each at 1/16, its channels spread over 4 x 4 pixels), the CNN features and the transformer features; then the regressor (two 3 x 3 convolutions, GELU between) and the condition the refinement starts from: [regressor out, picture blocks, features, probability].
Effects: fail, read, write, console, io, alloc.
Model.pointTransformer
function Model.pointTransformer(self: Model, x: Tensor, points: Tensor, views: Vec<View>, h: i64, w: i64, k: i64, at: string): TensorThe point transformer at (blocks of neighbor attention, each followed by a multi-view block where it has one) over the features
x ([v h w, c]) of the points ([v h w, 3]) of the views' h x w latent grids, with k neighbors a point: its output.
Effects: fail, read, write, console, io, alloc.
Model.gaussians
function Model.gaussians(self: Model, latents: Vec<Latent>, views: Vec<View>): Vec<Tensor>The first Gaussians' raw parameters ([v h w, 58]), the features the refinement starts from ([v h w, 512]) and the points: each latent pixel's condition through proj, plus the point transformer of that, through the Gaussian head (two linears, GELU between).
Effects: fail, read, write, console, io, alloc.
Model.drawn
function Model.drawn(self: Model, params: Tensor, views: Vec<View>, h: i64, w: i64, view: View, width: i64, height: i64): TensorThe picture ([3, height, width]) of the Gaussians params seen from view, as ReSplat draws it (black behind).
Effects: fail, read, write, console, io, alloc.
Model.refined
function Model.refined(self: Model, params: Tensor, state0: Tensor, seen: Vec<Seen>, views: Vec<View>, steps: i64): Vec<Tensor>steps rounds of refinement of the Gaussians params from the state the first pass left ([v h w, 512]), against
the views read alone (seen): each round draws every view, reads its error (ResNet features and the color
error of each latent pixel's block, through one multi-view block), runs the update's point transformer over the
Gaussians with their state and error, and adds its head's change. The Gaussians after each round.
Effects: fail, read, write, console, io, alloc.
Model.seen
function Model.seen(self: Model, pictures: Tensor): Vec<Seen>Each picture of pictures ([v, 3, H, W]) read alone.
Effects: fail, read, write, console, io, alloc.
Model.reconstruct
function Model.reconstruct(self: Model, seen: Vec<Seen>, views: Vec<View>, steps: i64): TensorThe Gaussians ReSplat makes of posed views read alone (seen) after steps rounds of refinement: their parameters
([v H W / 16, PARAMS]; splatsOf(..., H / LATENT, W / LATENT) draws them).
Effects: fail, read, write, console, io, alloc.
Model.world
function Model.world(self: Model, pictures: Tensor, views: Vec<View>, steps: i64, group: i64, stride: i64): WorldThe world of posed pictures ([v, 3, H, W]; views in one frame, in their order along the path): ReSplat over runs of
group views that start every stride views. On an open path the last run ends at the last view; on a closed one
(closed()) the runs go on around the end into the first views, so no seam is left there.
Effects: fail, read, write, console, io, alloc.
World.ply
function World.ply(self: World): bytesThe world as a 3DGS PLY file (gizai/render/gaussians' writePly): every run's splats in turn, and in the header a comment "run N" for each (its splats) followed by a comment "view" for each of its views: c2w (16 values), k (9), near and far. Any 3DGS viewer shows all runs at once; worldOf reads the runs back.
Effects: fail, read, write, console, io, alloc.
World.nearest
function World.nearest(self: World, view: View): TensorThe run of the world nearest view: the least distance from its middle view's center plus how far their forward
axes turn apart (1 - cos, times the runs' mean spacing).
Effects: fail, alloc.
Constants
CANDIDATES
const CANDIDATES: i64 = 128The depth candidates of the plane sweep, evenly spaced in log depth from near to far.
LATENT
const LATENT: i64 = 4The latent grid's step: a Gaussian a LATENT x LATENT block of pixels.
DEGREE
const DEGREE: i64 = 3The degree of ReSplat's colors.
PARAMS
const PARAMS: i64 = 59ReSplat's Gaussians as it refines them ([n, PARAMS] a Gaussian): the center (world), the scales, the rotation as an unnormalized quaternion x, y, z, w in its view's camera, the opacity before the sigmoid, and 3 x 16 color coefficients (r's, g's, b's, each from degree 0 up).