# gizlang:bench benchmarks as data. A suite declares its items (the rows of published tables), its tables (groups, with their method), its goals, and one function that measures the chosen items at a size. A suite program is `Bench.main(sys, suite)`; that is its whole command line: giz SUITE.giz list the items giz SUITE.giz run [ITEM...] [--size S] [--out FILE] giz SUITE.giz show the published results of the suite giz SUITE.giz compare [ITEM...] [--size S] a run beside the published values giz SUITE.giz record [ITEM...] a full run merged into the published file (a source checkout) ITEM is an item id, with or without the suite id before it. S is `quick` (seconds), `std` (the default) or `full` (the size that was published). A run keeps what it measured item by item, so a stopped run is still of use. A measure can run a program of any language: the program prints one line `benchITEMSUBJECTVALUE` for each value (`report` writes it), and `program` runs it and reads the lines (`reported`). A tool that is not on the machine skips its items with the way to get it (`needs`). A published file (`gizlang.bench.v1`, results/AREA/NAME.json in the published results folder) is the one place a measurement is written: the site, the READMEs and `compare` read it. A row holds the value of each subject (Gizlang and the tools it is compared with), when and where it was measured, and whether a program wrote it (`run`) or it was moved in from a page by hand (`migrated`). `text` is shown instead of the number when a cell is not a single number. ```gizlang import * as Bench from "gizlang:bench"; ``` ## Types ### `Subject` ```gizlang type Subject = { id: string; name: string; } ``` A tool that a suite measures: Gizlang, or one it is compared with. - `id`: `string` - `name`: `string` ### `Item` ```gizlang type Item = { id: string; group: string; title: string; unit: string; better: string; } ``` One row. `group` names the table that holds it ("" for the only table); `better` is "higher" or "lower". - `id`: `string` - `group`: `string` - `title`: `string` - `unit`: `string` - `better`: `string` ### `Value` ```gizlang type Value = { subject: string; value: f64; text: string; version: string; } ``` The value of one subject in one row. - `subject`: `string` - `value`: `f64` - `text`: `string` - `version`: `string` ### `Measured` ```gizlang type Measured = { item: string; values: Vec; why: string; } ``` The answer of a measure for one item: its values, or `why` it did not run. - `item`: `string` - `values`: `Vec` - `why`: `string` ### `Suite` ```gizlang type Suite = { id: string; title: string; subjects: Vec; groups: Vec; goals: Vec; items: Vec; measure: (sys: System, items: Vec, size: string) => Vec; } ``` A suite: its tables, goals and items, and how to measure the chosen items. `measure` answers one Measured for each item it was given; a subject that could not run has no value. - `id`: `string` - `title`: `string` - `subjects`: `Vec` - `groups`: `Vec` - `goals`: `Vec` - `items`: `Vec` - `measure`: `(sys: gizlang_system.System, items: Vec, size: string) => Vec` ### `Row` ```gizlang type Row = { item: Item; size: string; date: string; machine: string; build: string; source: string; note: string; values: Vec; } ``` A measured row: the item, the size, when, where, with which build, and the values. - `item`: `gizlang_bench.Item` - `size`: `string` - `date`: `string` - `machine`: `string` - `build`: `string` - `source`: `string` - `note`: `string` - `values`: `Vec` ### `Group` ```gizlang type Group = { id: string; title: string; method: string; } ``` A table of rows: `title` names it and `method` says how it was measured (Markdown, one paragraph). - `id`: `string` - `title`: `string` - `method`: `string` ### `Goal` ```gizlang type Goal = { area: string; text: string; item: string; } ``` A target: `area` names it, `text` states it, `item` (an item id, or "") says which row shows the progress. - `area`: `string` - `text`: `string` - `item`: `string` ### `Published` ```gizlang type Published = { suite: string; title: string; subjects: Vec; groups: Vec; goals: Vec; rows: Vec; } ``` The published file of one suite. - `suite`: `string` - `title`: `string` - `subjects`: `Vec` - `groups`: `Vec` - `goals`: `Vec` - `rows`: `Vec` ## Functions ### `ours` ```gizlang function ours(value: f64): Value ``` The subject of Gizlang's own value. ### `of` ```gizlang function of (subject: string, value: f64): Value ``` The value of another subject. ### `measured` ```gizlang function measured(item: Item, values: Vec): Measured ``` The values of ITEM. ### `skipped` ```gizlang function skipped(item: Item, why: string): Measured ``` ITEM did not run, for the reason WHY. ### `eachItem` ```gizlang function eachItem(sys: System, items: Vec, size: string, one: (sys: System, item: Item, size: string) => Result, string>): Vec ``` A measure of ITEMS that measures them one at a time with ONE. Effects: fail, read, write, console, io. ### `needs` ```gizlang function needs(sys: System, name: string, how: string): string throws string ``` The value of the environment variable NAME (a tool's path, a data folder), or why the items that need it skip: `HOW` says how to get it. Effects: fail, io. ### `wants` ```gizlang function wants(items: Vec, id: string): bool ``` Whether ITEMS has the item ID. Effects: fail. ### `report` ```gizlang function report(item: string, v: Value): void ``` Prints the line that tells a suite the value V of ITEM: `benchITEMSUBJECTVALUETEXTVERSION`. A program that a suite runs calls it once per value, beside its own output. Effects: fail, console. ### `reported` ```gizlang function reported(items: Vec, out: Result): Vec ``` The values that the lines of a program's output OUT report, as one Measured for each of ITEMS. When the program did not run (OUT is an error), every item skips with that reason. Effects: fail. ### `suites` ```gizlang function suites(sys: System): string ``` The folder that holds the suites (`bench` in a checkout); `GIZLANG_BENCH_SUITES` (set by `giz bench`) overrides it. Effects: fail, io. ### `program` ```gizlang function program(sys: System, path: string, args: Vec, environment: Vec): string throws string ``` Runs the Gizlang program PATH (relative to the suites folder) with ARGS and the added ENVIRONMENT, shows what it prints, and answers its output, or why it failed. Effects: fail, console, io. ### `cli` ```gizlang function cli(sys: System): string ``` The command that runs Gizlang programs (`giz`); `GIZLANG_CLI` (set by `giz bench`) overrides it. Effects: fail, io. ### `run` ```gizlang function run(sys: System, program: string, args: Vec, environment: Vec): string throws string ``` Runs the command PROGRAM with ARGS and the added ENVIRONMENT, shows what it prints (the report lines aside), and answers its output, or why it failed. Effects: fail, console, io. ### `scaled` ```gizlang function scaled(size: string, quick: i64, std: i64, full: i64): i64 ``` A number that fits SIZE: QUICK for `quick`, FULL for `full`, STD otherwise. ### `nsPerCall` ```gizlang function nsPerCall(count: i64, work: (i: i64) => i64): f64 ``` Nanoseconds per call of WORK over COUNT calls. WORK answers a number; the numbers are summed so a call cannot be dropped as unused. Effects: fail, read, write, console, io. ### `figures` ```gizlang function figures(value: f64): string ``` VALUE as a person reads it: three figures, no trailing zeros, thousands separated. Effects: fail. ### `format` ```gizlang function format(value: f64, unit: string): string ``` VALUE with its UNIT: `302,589 req/s`, `21 ms`, `53%`. Effects: fail. ### `shown` ```gizlang function shown(v: Value, unit: string): string ``` What a cell shows: its `text` when it has one. Effects: fail. ### `encode` ```gizlang function encode(p: Published): string ``` The text of a published file (two-space indent, so a diff shows one row per change). Effects: fail. ### `decode` ```gizlang function decode(text: string): Published throws string ``` The published file in TEXT. Effects: fail. ### `merged` ```gizlang function merged(base: Published, run: Published): Published ``` BASE with the rows of RUN: a row of an item that RUN measured is replaced in its place, a new one is added at the end, the others stay. The title, subjects, groups and goals are RUN's: the suite program owns them. Effects: fail. ### `machine` ```gizlang function machine(sys: System): string ``` The machine a run is on, as a published row states it: `AMD Ryzen 9 7945HX (32 threads), 64 GB, Linux 6.17`. Effects: fail, io. ### `published` ```gizlang function published(sys: System, dir: string, id: string): Published throws string ``` The published file of the suite ID in DIR (the folder that holds AREA/NAME.json), if there is one. Effects: fail, io. ### `main` ```gizlang function main(sys: System, suite: Suite): i64 ``` The command line of a suite program. Answers the exit status. Effects: fail, read, write, console, io. ## Methods ### `Row.value` ```gizlang function Row.value(self: Row, subject: string): Option ``` The value of SUBJECT in the row, if the row has one. Effects: fail. ### `Published.row` ```gizlang function Published.row(self: Published, id: string): Option ``` The row of item ID, if P has one. Effects: fail. ### `Published.name` ```gizlang function Published.name(self: Published, subject: string): string ``` The name of SUBJECT in P. Effects: fail. ### `Published.group` ```gizlang function Published.group(self: Published, id: string): Group ``` The group ID of P (its title and method); an empty group when P has none of that id. Effects: fail. ## Constants ### `SCHEMA` ```gizlang const SCHEMA = "gizlang.bench.v1" ``` The schema name that each published result file carries.