gizai/world/scene
a 3D scene of one photo (Mira-Scene's task) on gizlang's own parts. A vision chat model names the things, their boxes and what holds each; Pi3X gives each pixel's point in meters; the cutter cuts each thing out; a mesh model that sees the photo's camera (Pixal3D: its mesh is aligned to the picture) makes each thing, rigged when it is a character. So a thing's place needs no model of canonical coordinates (Mira-Scene's CCM): its size and place follow in closed form from its crop, its depth and the focal length. A plane fitted to the floor's points gives the up direction; things stand on the floor or on the surface of what holds them that is nearest under their measured place (a shelf, a top); the floor is a plane colored from the photo, and the rest of the photo is the backdrop, a mesh of its points that goes on behind the things.
The cutter cuts out what stands out in a crop, which is not always the thing named: a stool before a desk is the desk's cut too. So each pixel is one thing's (claimed): a cut under two names is the thing of the smaller box, a thing inside a larger thing's cut at its depth is a part of that thing (books on a shelf: made with the shelf), and a thing before another takes its pixels from it.
import * as Scene from "gizai/world/scene";Types
Found
type Found = { name: string; box: Vec<f64>; on: i64; character: bool; }A thing the vision model found: its name, its box (x0, y0, x1, y1 in the photo's pixels), what holds it, and whether it is a character (rigged).
name:stringbox:Vec<f64>on:i64character:bool
Parts
type Parts = { look: (photo: Img.Image) => (() => string throws string) throws string; points: (photo: Img.Image) => Points throws string; cuts: (crops: Vec<Img.Image>) => (() => Vec<bytes> throws string) throws string; sculptor: () => Sculptor throws string; }The parts a scene is made with: the vision model's answer to LOOK_PROMPT about a photo, the photo's points
(Pi3X), the cutter (a picture's alpha) and the mesh model (a Sculptor with mesh). Each may load its model when called
and free it after, so that one card of 16 GB holds them in turn.
look:(arg0: gizlang_image.Image) => (() => string throws string) throws string— Starts the look at the photo; what it gives waits for the answer. A look on another machine runs beside the work here.points:(arg0: gizlang_image.Image) => M79.Points throws stringcuts:(arg0: Vec<gizlang_image.Image>) => (() => Vec<bytes> throws string) throws string— Starts cutting the things out of their crops; what it gives waits for each crop's alpha. Cuts on another machine run beside the work here.sculptor:() => M86.Sculptor throws string
Functions
foundOf
function foundOf(text: string, width: i64, height: i64): Vec<Found> throws stringThe things of the vision model's answer text (LOOK_PROMPT's list) in a photo of width x height. A model
may wrap the list in an object ({"items": [...]}), in a code fence, leave the ids out (they count from 1), and
name a holder by its id (a number, or its digits) or by its name (of two things of the name, the one whose box
lies the most under the thing's).
Effects: fail, alloc.
scene
function scene(parts: Parts, photo: bytes, seed: i64, progress: (stage: string, step: i64, total: i64) => bool): bytes throws stringThe scene of photo (a JPEG, PNG or WebP file's bytes) as one GLB: the backdrop, the floor, and each thing (named
by what it is; a character rigged), in meters, y up, the floor at y = 0, the photo's camera at x = z = 0 looking
toward -z. progress hears the stage, the step and the steps; false from it stops the work.
Effects: fail, read, write, console, io, alloc.
Constants
LOOK_PROMPT
const LOOK_PROMPT = "List every separate movable thing in the picture: furniture, objects, people and animals. " + "Do not list the floor, walls, ceiling, windows, doors or the sky. Answer with JSON only, no other words: " + "[{\"id\":1,\"name\":\"a short name\",\"box\":[x0,y0,x1,y1],\"on\":\"floor\" or \"wall\" or the id of the " + "thing it stands on or in,\"character\":true for a person or an animal}]. Ids count from 1. Boxes are on a 0 to " + "1000 scale of the picture's width and height."What the vision model is asked about the photo.
ON_FLOOR
const ON_FLOOR: i64 = -1What holds a thing: the floor, nothing (a wall, the ceiling, or a part of the room that is no thing), or another thing (its index among the found, 0 or more).
ON_NOTHING
const ON_NOTHING: i64 = -2A thing held by nothing: on a wall or the ceiling.