GizStackGizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/world/scene

a 3D scene of one photo (Mira-Scene's task) on gizlang's own parts. A vision chat model names the things, their boxes and what holds each; Pi3X gives each pixel's point in meters; the cutter cuts each thing out; a mesh model that sees the photo's camera (Pixal3D: its mesh is aligned to the picture) makes each thing, rigged when it is a character. So a thing's place needs no model of canonical coordinates (Mira-Scene's CCM): its size and place follow in closed form from its crop, its depth and the focal length. A plane fitted to the floor's points gives the up direction; things stand on the floor or on the surface of what holds them that is nearest under their measured place (a shelf, a top); the floor is a plane colored from the photo, and the rest of the photo is the backdrop, a mesh of its points that goes on behind the things.

The cutter cuts out what stands out in a crop, which is not always the thing named: a stool before a desk is the desk's cut too. So each pixel is one thing's (claimed): a cut under two names is the thing of the smaller box, a thing inside a larger thing's cut at its depth is a part of that thing (books on a shelf: made with the shelf), and a thing before another takes its pixels from it.

gizlang
import * as Scene from "gizai/world/scene";

Types

Found

gizlang
type Found = { name: string; box: Vec<f64>; on: i64; character: bool; }

A thing the vision model found: its name, its box (x0, y0, x1, y1 in the photo's pixels), what holds it, and whether it is a character (rigged).

  • name: string
  • box: Vec<f64>
  • on: i64
  • character: bool

Parts

gizlang
type Parts = { look: (photo: Img.Image) => (() => string throws string) throws string; points: (photo: Img.Image) => Points throws string; cuts: (crops: Vec<Img.Image>) => (() => Vec<bytes> throws string) throws string; sculptor: () => Sculptor throws string; }

The parts a scene is made with: the vision model's answer to LOOK_PROMPT about a photo, the photo's points (Pi3X), the cutter (a picture's alpha) and the mesh model (a Sculptor with mesh). Each may load its model when called and free it after, so that one card of 16 GB holds them in turn.

  • look: (arg0: gizlang_image.Image) => (() => string throws string) throws string — Starts the look at the photo; what it gives waits for the answer. A look on another machine runs beside the work here.
  • points: (arg0: gizlang_image.Image) => M79.Points throws string
  • cuts: (arg0: Vec<gizlang_image.Image>) => (() => Vec<bytes> throws string) throws string — Starts cutting the things out of their crops; what it gives waits for each crop's alpha. Cuts on another machine run beside the work here.
  • sculptor: () => M86.Sculptor throws string

Functions

foundOf

gizlang
function foundOf(text: string, width: i64, height: i64): Vec<Found> throws string

The things of the vision model's answer text (LOOK_PROMPT's list) in a photo of width x height. A model may wrap the list in an object ({"items": [...]}), in a code fence, leave the ids out (they count from 1), and name a holder by its id (a number, or its digits) or by its name (of two things of the name, the one whose box lies the most under the thing's).

Effects: fail, alloc.

scene

gizlang
function scene(parts: Parts, photo: bytes, seed: i64, progress: (stage: string, step: i64, total: i64) => bool): bytes throws string

The scene of photo (a JPEG, PNG or WebP file's bytes) as one GLB: the backdrop, the floor, and each thing (named by what it is; a character rigged), in meters, y up, the floor at y = 0, the photo's camera at x = z = 0 looking toward -z. progress hears the stage, the step and the steps; false from it stops the work.

Effects: fail, read, write, console, io, alloc.

Constants

LOOK_PROMPT

gizlang
const LOOK_PROMPT = "List every separate movable thing in the picture: furniture, objects, people and animals. " + "Do not list the floor, walls, ceiling, windows, doors or the sky. Answer with JSON only, no other words: " + "[{\"id\":1,\"name\":\"a short name\",\"box\":[x0,y0,x1,y1],\"on\":\"floor\" or \"wall\" or the id of the " + "thing it stands on or in,\"character\":true for a person or an animal}]. Ids count from 1. Boxes are on a 0 to " + "1000 scale of the picture's width and height."

What the vision model is asked about the photo.

ON_FLOOR

gizlang
const ON_FLOOR: i64 = -1

What holds a thing: the floor, nothing (a wall, the ceiling, or a part of the room that is no thing), or another thing (its index among the found, 0 or more).

ON_NOTHING

gizlang
const ON_NOTHING: i64 = -2

A thing held by nothing: on a wall or the ceiling.