GizStack
View .mdOpen in ChatGPTOpen in Claude

Measure and improve performance

This guide shows how to find where a program's time and memory go, why, and how to make it faster and smaller. One tool answers these questions for every Gizlang program, the compiler included: giz profile. It needs no perf, no privileges, no install and no special build. Its answers are JSON, for a program or an agent to loop on. Published numbers and how to reproduce them are in Benchmarks.

The method

  1. State the question. "Why does a page load take 4 s?" is a question. "Make it faster" is not.
  2. Profile once, small. Run the smallest input that shows the cost, for a few seconds.
  3. Find the cause on the critical path. explain ranks the causes by what removing each saves at most. CPU off the critical path costs CPU, not time: making it faster does not shorten the run.
  4. Test the hypothesis. Each cause comes with an experiment. Change one thing, record again, and compare: keep the change only when its interval excludes zero.
  5. Measure the whole once, at the end. While changes still improve, measure the rows in question only. Then run the full benchmark once and report the best of 3 runs.

When two attempts at the same failure do not find its cause, stop guessing. Get more visibility first: a span, a phase, a narrower --focus, a query of the records.

Generated frame names are resolved from the executable's program table and 64 KiB name windows. Symbolication does not read the complete executable into memory. Keep a workload's memory scope inside giz profile run when the profile reader itself needs a different limit; the recorded workload still runs under its own limit. Comparisons require at least one eligible warm run on each side. A side with only excluded cold runs is an error, never a zero-cost measurement.

Record a session

sh
giz profile capabilities                         # what this host records, how exactly, and what is missing
giz profile run -- giz check main.giz            # record a command in the foreground
giz profile run --repeat 5 -- ./bench            # five runs; compare sets cold ones apart by their evidence
giz profile run --repeat 5 --alternate "./old" -- ./new  # old and new runs in turn, then their comparison
giz profile run --until listening -- ./server    # a server's start, up to its ready line
giz profile start -- ./server --port 8080        # record a server that outlives this command
giz profile start --paused -- ./server           # ... recording only from `attach` on
giz profile attach latest                        # (and `pause`)
giz profile stop latest                          # end the recording, then the program (--keep: not the program)
giz profile status latest                        # recording state and every process
giz profile list                                 # recorded sessions
giz profile gc --days 7                          # collect old sessions

A session is a private folder (mode 0700) under $GIZLANG_PROFILE_DIR/ID, by default $GIZLANG_CACHE/profile/ID: session.json, a folder for each run, the program's output in output.log, and a file for each process (mode 0600: stacks and arguments may be private). latest names the newest, and a view with no session reads it. A session's runs record every process the program starts (GIZLANG_PROFILE=DIR/%p): a test runner, the compiler, workers, servers it starts, also through a program that is not Gizlang (env, a shell). Each recording adds its start time and instance to the PID filename, preserving successive program images across exec. CPU totals cover each image separately; causal paths retain the predecessor. --limit-bytes N stops a process's recording at N bytes (1 GiB by default), and says so. Each new session first removes the sessions that ended 7 days ago and ends the recording of older running ones; gc --days N does it now. While run runs, each finished run prints a progress line to standard error ({"progress":{"session":...,"run":K,"of":N,"exit":...,"wallNs":...}}). When both standard input and output are terminals, a foreground run keeps them for the program, so a TUI stays interactive; its errors go to output.log and the envelope follows its exit. Piped and detached runs log both outputs and print only the envelope. To record an interactive program, run giz profile run -- giz desk --terminal directly in the terminal without a pipe or redirect.

A host without giz records too: start the program there with GIZLANG_PROFILE=DIR/profile.%p (and GIZLANG_PROFILE_GPU=kernels for each kernel's time), copy DIR, and give DIR to any command as the session.

A recorded giz build or giz test runs on this machine, in one of its local build slots. On a build host, record there: GIZLANG_PROFILE=DIR/%p goes with the command to the host, and giz profile explain DIR in the slot's tree reads it. A profile holds Gizlang processes; the launcher (bin/gizlang, a shell script) is not one. When a command is slow and its processes are not, time the launcher's steps: PS4='+ $EPOCHREALTIME ' bash -x bin/gizlang COMMAND 2> trace, then read the gaps between the lines.

A setting that changes how the program runs goes on its command, not on giz: giz profile run -- env GIZLANG_WORKERS=4 ./server. Every Gizlang program reads it, so on giz it would fork giz itself.

Session commandResult
status SSession state, whether its program lives, and each recorded process's end, samples and losses.
listSessions, newest first, with their commands, runs and state.
gc [--days N]Remove ended sessions older than N days (7 by default) and end older recordings still running.

What a run records:

  • CPU. A timer on each thread's own CPU clock samples it about 1000 times a second of its CPU (--hz; a kernel checks the timers at its tick, so CONFIG_HZ=250 gives at most 250). A sample stands for the CPU its thread used since the previous one, so the tables add up to the CPU time of the run. Each sample carries its thread's task and span, its page faults and context switches, and its cycles, instructions, cache misses and branch misses when the kernel allows them (perf_event_paranoid 2 or less). A parked thread costs nothing. A thread that runs the VM (gizvm: JavaScript, Python) has its guest frames too: each function shows as vm NAME (FILE) in the stack, where the interpreter runs it, or outermost when compiled guest code cut the native stack. flame --focus "vm " and top --sort inclusive --focus "vm " rank them.
  • Tasks. The scheduler records each task's spawn (with its site), each run on a thread and why it stopped (it yielded, parked for I/O, a join, a sleep, a channel or a thread, or ended), each wake and what woke it: the I/O operation whose completion woke it, or the task. Parallel work on pool threads counts as its caller's task.
  • I/O. Each operation of the I/O loop: its kind (file, socket, DNS, timer, fdatasync, pool work), descriptor, bytes, status, submit and completion, and the time a pool operation waited for a pool thread. A stream's reads and direct writes are summed a millisecond at a time. /proc/self/io says how much of the reading came from the device, and /proc/diskstats what the devices did.
  • Network. Each TLS handshake with its host and result, TCP_INFO at each connect, accept and close (round trip, retransmissions, delivery rate, receive-window and send-buffer limits), and when the peer acknowledged the bytes a write sent: a write ends when the kernel has the bytes, an acknowledgement when the peer has them, and a response is done when the program says so.
  • GPU. Each kernel, graph replay and copy of gizhw/gpu, timed on the GPU between two events that are read back without a sync between launches; their CPU submit, their mapping onto the CPU's clock with its error, the device memory, each device buffer with the stack that asked for it and when it was dropped, each time the CPU waited for the GPU, and the work a launch declares (Gpu.work: its operations and bytes moved), which gives each kernel's TOP/s and GB/s to set beside the device's peaks (attention and matrix products declare theirs). A graph replay is one record; with GIZLANG_PROFILE_GPU=kernels, Gpu.profiling is on, and a program that records only when it is off (gizai's engine) runs its kernels one by one, each timed (slower: for where the time goes, not for how fast). Each kernel's code (PTX, LLVM IR) is kept beside the recording with its registers, spilled bytes, workgroup memory and the workgroups that fit on a unit at once, and each tune keeps every choice's time (kernels). The measuring of a kernel's choices and each compilation are the spans gpu tune and gpu compile: explain sums them as the cause gpu:first-run, the time a process pays once, which a process that learned the kept choices does not. Its cause gpu:kernels names the kernels with the largest share of the GPU's time, each with its TOP/s. The driver's allocation, freeing and launch calls are the spans gpu alloc, gpu free and gpu launch (bytes and status in each): a driver's library has no frame pointers, so its samples end in ioctl with no caller, and query S --kind span --where name~gpu names the call and sums its CPU time. Each launch keeps the stack that asked for it: resource S gpu gives the GPU's time by that code (callers: the first function past the GPU and the tensor engine, then its caller), so a model's blocks, norms and attention show under their own names. The files kept beside a recording are written without disk syncs.
  • Spans. Code names its requests and steps with gizlang/profile (below). An HTTP server makes a span of each request and joins the caller's span from its traceparent; an HTTP client sends its running span. GizDB makes a span of each statement, and the LLM engine of each prefill and decode step.
  • Phases. A worker (gizlang/worker) marks each stage a job reports as a phase, and idle at its end; giz ai marks the stages of its image, video and object commands. phases S gives each stage's time, and --phase generating restricts every view, the GPU's too.
  • Processes. Each process writes its own file with a random instance id; a child names the span that started it, so the processes of a run form one graph. Two workers of one program never mix.
  • Waits. A thread off the CPU for 10 ms or more: what it waited for and the stack it waited in, an estimate from the kernel's wchan each 10 ms. A task's waits are exact (above). Each running thread's run-queue delay comes from its schedstat.
  • Children. Each process the program started and reaped: its program, arguments, run time and exit.
  • Memory. The resident bytes every 10 ms; the owners of the memory (below), and with more than one NUMA node the pages on each (numa_maps), when it grew by a quarter and 64 MiB, at each phase and at exit. Such a reading walks the page tables and holds the lock the program's mmap waits for, so it comes no sooner than 100 times its own cost after the last one. The pages each sample's thread touched first (minor faults) and waited for the disk for (major faults); and a malloc block per 512 KiB given on average with its stack and its free, so the sums are unbiased.
  • Quality. At exit each process records its CPU time, the recorder's own (image preservation, its sampler thread, its signal handler and its events), what was lost or cut, its exit status, and the cores Task.cores() gave it beside those its affinity and quota allow and the machine has. Each file a process runs code from is kept once in GIZLANG_PROFILE_FILES (relative to the profile's folder; files by default, a session's files for all its runs), so a program deleted or rebuilt after the run is still named. Images are preserved before events start, so a slow initial copy cannot fill the event rings while holding the recorder's writer lock. This preparation adds startup time and counts as recorder overhead.

The recorder's cost is measured on each thread's CPU clock and reported in quality.overheadPpm. On 16 CPUs, with the same work split over 1 to 256 threads, the CPU with and without recording differed by less than the runs did (5%); 1000 threads that each lived 1.5 ms took up to 12% more CPU at 997 Hz, as each new thread gets its timer and ring. A sample or an event goes into its thread's ring in memory, which the sampler thread writes out; no record makes a system call to the file, takes a lock or waits. An event that finds its ring full writes the ring out itself when no other thread is writing; otherwise the record is dropped and counted.

Ask the profile

resource S gpu --kernels includes each kernel's minimum, median (p50Ns), nearest-rank 95th percentile (p95Ns) and maximum recorded duration beside its mean. Use the spread to see clock ramp-up and outliers before comparing small kernels. HIP and CUDA report resident workgroups per compute unit and the device's unit count in each kept kernel's usage. LLM GEMV and GEMM launches declare useful operations and minimum input/output bytes. Their rates show useful TOP/s and minimum GB/s, not measured DRAM bandwidth: caches, tiled rereads and split sums change physical traffic. Graph recordings keep the sum of their launches' declared work. Replay rates use two timing events for a whole graph, so individual event pairs do not perturb each small kernel. Compare these with a separate kernel profile. Its staticCode counts the instructions, global and shared memory operations, and workgroup barriers in all kept code, including the prologue and epilogue that a loop-only comparison misses. These are static counts, not executed instruction counters: branches, unrolling and loop trip counts change how many run. A kernel comparison's whole reports instruction changes by kind over all the code, including kernels without loops; its top-level instructions and ops describe the hot loop. Read code and source to check whether both sides of a comparison use SASS, PTX or LLVM IR.

Source snapshots contain the code selected for compilation, including a GIZLANG_GPU_SOURCES override. The SASS cache names its PTX content hash in a header; a changed PTX or a cache without that header is rebuilt before analysis.

Every command prints one JSON envelope (--text renders the same answer for a person), and its exit code is 0 or that of its error (2 usage, 3 regression, 4 data, 5 not found, 6 unsupported, 7 the program failed):

json
{"schema":"gizlang.profile/2","command":"explain","ok":true,"session":"20261107-153012-ab12",
 "quality":{"complete":true,"processes":3,"ended":3,"coverage":97,"samples":2210,"events":18321,"lost":0,
            "overheadPpm":14000,"warnings":[]},
 "data":[...],"page":{"cursor":"100","more":true},"next":["giz profile critical-path 20261107-153012-ab12"]}

quality says what the data is worth: complete is false when a process has no end record (killed, or still running), a file was cut or reached its limit, records were lost, or the samples cover under 90% of the CPU, and each warnings line names the loss. A number from incomplete data is a lower bound. next holds the commands that look closer; page the cursor of the next rows (--cursor, --limit).

CommandAnswers
explain S [--target T]Rows of {run, cause}, largest cost first. Each cause has what removing it saves at most (costNs, lowNs..highNs), how sure that is (measured, sampled, estimated), its evidence (record ids), a falsifiable hypothesis, the experiment that tests it, and the commands that look closer. A task that ended with a fault comes first (kind: fault): its claim names the task and the fault, its hypothesis is the fault's report.
critical-path S [--target T]The chain of work that set the target's end: each stretch's kind (cpu, queue: runnable but its thread ran other work, io, pool: waiting for a pool thread, dispatch, signal, sleep, gpu, blocked, spawn, unknown; inside a child: the path of its own recording when it is Gizlang, else child-cpu, child-queue and child-wait, measured from its scheduler counters at each 10 ms look, and child where no look covers it), its time, task, what it was and its records; the time of each kind; and over many targets (the runs of a repeated session, each its own target, or the spans of a route) their p50, p90, p95 and p99 and each kind's p50, p95 and p99 (spread).
query S [--kind K] [--where F=V] [--from MS] [--to MS] [--sort S]The records: runs, wakes, spawns, io, spans, kernels, waits, tcp, acks and the rest, filtered (=, !=, >, <, >=, <=, ~ holds) and paged; --where id=eN reads one record that explain cites. --jsonl gives a row a line.
top S [--measure cpu|wall|memory|heap|leak|device] [--focus NAME] [--phase P] [--lines]The functions by their own and inclusive share, each with its sampling noise, IPC and cache misses, and with --lines its hottest source lines (from the program's debug information). With --focus F it also lists who called F and what F called, each with its share (--depth N: every function within N calls under F, once a stack). --per OPS adds the cost of one operation: OPS is a count, span:TEXT or mark:TEXT (GizDB makes a span of each statement, so --per span:SELECT gives the CPU of one SELECT). leak ranks the code whose malloc blocks a process never freed before it ended.
flame SFolded stacks with their weight; --text prints a stack table.
resource S ioEach kind of I/O operation (file read, fdatasync, ...; by the kernel's ring or the pool apart), the time pool operations waited for a pool thread, each process's bytes and the share the page cache served, and each device's busy share and queue. children groups child programs by run count, summed and overlapping wall ns, peak concurrency and longest invocation.
resource S netEach socket operation, TLS handshake and DNS lookup, how long the peer took to acknowledge sent bytes, the HTTP requests by route (their spans), and each socket's round trip, retransmissions, delivery rate and the time the receiver's window or the send buffer limited it.
resource S gpuThe kernels, graph replays and copies by name, the kernels and replays by the span that submitted them (llm decode, llm prefill, a request), the time from each submit to its start on the GPU, the CPU's waits for the GPU, each stream's busy time, how much the streams overlapped, the device memory at its most, the clock mapping's error, and each kernel's rates (rates: TOP/s and GB/s of the work its launches declared).
resource S gpu --kernels [--kernel TEXT] [--against S2] [--against-kernel TEXT2]Each GPU kernel the runs kept, by GPU time: launches, mean time, registers, spilled (local) bytes, workgroup memory, workgroups at once on a unit and the units, its code, and its hot loop (the most matrix instructions, then fused multiply-adds, then instructions) with its instructions by kind, its global loads a turn (loads) and the most of them in flight (inFlight: issued and not yet read, across the branch back too; one or two leave memory idle while each load waits); and what each tune measured, each choice's fastest run and the one taken. --kernel gives every loop of the matching kernels. --against compares the hot loop of a kernel of another session (or, with --against-kernel, another kernel) with this one's, by kind of instruction: what a code change did to the loop. On NVIDIA the loops are machine code (SASS): the PTX compiled by ptxas -O3 for its target and read by nvdisasm, where a CUDA toolkit is on PATH, in CUDA_HOME or under /usr/local/cuda* on the host that reads the session (kept as NAME-HASH.sass). It is the driver's compiler; a toolkit of another version may schedule a little differently. Without a toolkit the loops are the PTX, which the driver still unrolls and schedules (code says which).
resource S memoryEach process's resident bytes at the most and at the end, its owners and NUMA nodes at the most, the malloc bytes held at the most and never freed by stack, how long freed blocks lived by function, and the device buffers held at their most by stack.
timeline S [--bucket MS]Each slice: CPUs busy, threads, tasks running, I/O, file and network operations in flight, GPU busy, the largest resident reading and the hottest function. resident is absent when that slice has no memory reading. GPU busy is wall time covered by at least one recorded device interval, with overlapping streams counted once. File/network counts include work without a named span; they describe concurrent operations, not device utilization. The next-page command preserves the bucket, run and time filters.
compare BASE NEW [--measure wall|cpu|function:NAME]The change of the mean with its 95% interval (Welch's t over the runs) and Hedges' g, each side's median, p95 and p99, the runs far from the rest (beyond 3.5 median absolute deviations, with the command that reads them), and each function's change. --phase, --from, --to and span --target select the same time windows as other views, for both compare and regress; wall time is their union, excluding gaps and counting overlaps once. Cold-cache and other-process evidence still describes the whole recording. A run is cold by its evidence, never by its place: most of its reads came from a device (/proc/self/io), or it is a first run beyond 3.5 median absolute deviations of the others; cold runs are compared apart, each with its reason (coldBefore, coldAfter). Each run records the machine around it (load, CPUs, clock, governor, the CPU the machine was busy), and a run during which other processes used more than half a CPU is named (disturbedBefore, disturbedAfter). With one run a side, the interval is the sampling noise alone, and says so.
regress BASE NEW --fail-if PCTExit 3 when the measure grew by more than PCT percent with its whole interval: a check for a script or CI.
export S --format chrome|pprof|otlp|folded|jsonl [--redact] [--out FILE]A Chrome trace (Perfetto opens it: tasks, I/O, spans and GPU streams on their tracks), a gzipped pprof profile of the samples (go tool pprof, and every tool that reads pprof; --measure picks its value), OpenTelemetry traces as OTLP JSON (the spans, I/O, children and GPU work, under the trace of the root span), folded stacks, or the records as JSON lines. --redact leaves out every text; --out - writes the file to standard output and the envelope to standard error.
phases SEach phase's process, start and wall ns, CPU and wait microseconds, faulted bytes and resident bytes at its start and most. A missing resident reading is absent.

A target (--target) is process:N (a process's whole run; process 0 by default), span:TEXT (every span whose name holds TEXT: each HTTP request of a route, each statement), or a span's record id. A span target narrows every view to the time of those spans in their process: top S --target "span:db select SELECT sum" is the CPU of those GizDB statements, with no times copied by hand. --run N reads run N of a session alone (1 is the first run; coldness comes from evidence). An invalid run is a usage error. Every row of a resource group has its count, total, p50, p95, p99, the longest, the bytes, the most in flight at once and the ids of its slowest records, for query.

The loop of an agent: run or start, explain, read the first cause's evidence with query and critical-path, change the code as its experiment says, run again, then compare the two sessions.

Text and selection

Append --text to the view you need: explain S --text, top S --measure wall --text, resource S memory --text, timeline S --bucket 20 --text, or phases S --text. Text renders the same envelope: data, units, quality, warnings, evidence and next commands. It does no analysis of its own. --text cannot combine with --jsonl or --out -, which are streaming formats.

--phase NAME, --target T, --from MS and --to MS select the data before a view computes its answer. Time is relative to the recording's first record. Use --run N to inspect a particular run; views sum the selected recordings by default. The causal commands analyze each run separately: critical-path reports a latency distribution and explain labels every cause with its run. Their quality covers all selected runs. A repeated phase keeps separate windows, each ending at its own last task. A wait or event crossing a window counts by its part inside it. Unknown flags, phases, targets, sort orders and run numbers fail explicitly. Each flag is accepted only by the commands that use it, as declared in the CLI command table. For example, top --where, status --run and explain --cursor are usage errors. Stack export options (--measure, --focus) require --format folded or pprof; run --fail-if needs a baseline or an alternate command.

GPU timing and code share one entry: resource S gpu for recorded work, resource S gpu --kernels for kept code and tuning, then --kernel TEXT or --against S2 to inspect or compare kernels. Code inspection covers the selected runs' kept kernels; phase and time filters apply to recorded work only. Combining --kernels with those filters is a usage error. --kernel and --against require --kernels.

Spans

A span names a request or a step of the program, and ties to it the tasks it spawns, their I/O and CPU, and the processes it starts. When the program is not profiled, each call costs one test.

gizlang
import {begin, counter} from "gizlang/profile";

type Request = { body: string; };
type Response = { status: i64; body: string; };

function work(request: Request): Response { return { status: 200, body: request.body }; }

export function handle(request: Request, waiting: i64): Response {
    const span = begin("checkout");
    const answer = work(request);
    counter("queue depth", waiting);
    span.finish(answer.body.length, answer.status, 0);
    return answer;
}

join(name, parent) starts a span under a span of another process (the 16 hex digits of a traceparent), and traceSpan() gives the running span to send. giz profile critical-path S --target span:checkout then gives each request's critical path and their latency distribution.

Phases

A program names the parts of its run with gizlang/profile. A phase lasts until the next one or the end. When the program is not profiled, phase costs one test.

gizlang
import {phase} from "gizlang/profile";

export function main(): i64 {
    phase("parse");
    let total = 0;
    for (let i = 0; i < 1000; i += 1) { total += i; }
    phase("report");
    console.log(`${total}`);
    return 0;
}

giz profile phases S then gives each phase its time and memory, and --phase report restricts top and every view to it. A wait, a child or an event that crosses a phase's start or end counts in it by the part inside it. A phase name that no process marked is an error, with the names that were marked. The GizBrowser command line marks load, render (which waits for the pictures) and draw.

Memory

Three questions, three answers:

  • How much, and when? timeline S shows resident bytes over time; resource S memory shows each process's peak and end. Outside a profile, sys.usage() gives a process's resident and peak bytes.

  • Who holds it? resource S memory lists the owners when the process was most resident, by proportional resident bytes:

    OwnerWhat it is
    gizlang pagesThe runtime's pages of small blocks: values, buffers and heap objects (Memory).
    gizlang large blocksBlocks past 64 KiB: large buffers, and freed ones kept for reuse.
    task stacksThe stacks of tasks.
    vm heap, vm large range, vm code, ...The JavaScript and Python VM's heap, its large objects and its compiled code.
    mallocMemory that no one named: the C library's blocks, the VM's compiled functions and other C code.
    file NAMEThe pages of a file it maps, such as the program itself.

    Memory.census(top) (gizlang/memory) splits the runtime's part further: its heap objects by type and its blocks by size class. A fault at the memory limit prints it.

  • Who made it grow? top S --measure memory ranks the functions by the memory they touched first.

  • Who holds the malloc part? top S --measure heap ranks the code that asked for the blocks held at the most.

To find a program's real working set, lower its limit until it fails: GIZLANG_MEMORY_LIMIT=150000000 PROGRAM. Under a limit, the allocator gives back what it keeps before it refuses. A run that passes at a limit far below its usual peak held memory that it did not need.

Instrumentation API

gizlang/profile exposes the whole instrumentation surface:

APIPurpose
recording(), kernels()Whether recording is active, and whether individual GPU graph kernels must be timed.
now(), newId(), current()Monotonic ns, a process-local event id (0 when off), and the running task's span id.
phase(name), label(at, size, name)Name a phase or a mapped memory range. Memory labels require Linux named anonymous mappings.
begin(name), join(name, parent)Start a span, locally or under another process's 16-hex-digit span id.
Span.end(), Span.finish(bytes, status, resource)End a span and restore its parent; optionally record its result. End nested spans in reverse order.
mark(name), counter(name, value)Record an instant or a named integer measurement.
traceSpan(), traceId()The running span's 16 hex digits (empty without one) and this process's 32-hex-digit trace id.
guest(vm, links, depth, room, exits, callee, info, kind, doubles)Install a VM stack layout for the calling thread's samples; return the previous VM address. Pass 0 to clear it.
guestName(info, name), guestCode(at, size, name)Name a guest function's metadata and its compiled code range.
event(kind, start, end, span, parent, resource, bytes, status, a, b, text)Emit a wire event for an integration such as GPU work. Use the exported event constants; do not copy wire numbers.

Measure a change

Pick the cheapest measure that answers the question. Make one change at a time, and revert a change that does not move the measure.

  • A session a side. giz profile run --repeat 5 records each side; compare BASE NEW gives the change with its interval and effect size, the cold first run apart. giz profile run --repeat 5 --baseline BASE --fail-if 5 -- PROGRAM records the new side and judges it in one call (exit 3 when it regressed).
  • Instructions (valgrind --tool=callgrind) are stable on a shared machine. Measure the margin: run two lengths and subtract, so that start-up is not spread over the work. For HTTP: bin/gizlang bench/web/http/run.giz instructions PORT COMMAND....
  • CPU time per unit of work comes from /proc/PID/stat of one pinned worker, or from a profile.
  • Wall-clock rates confirm the other two. Alternate both sides in the same period on pinned, separate CPUs, take the best of several runs, and make sure that the load generator is not the bottleneck.
  • System calls: strace -c.

A release build for an A/B

Compare release builds, not debug builds. giz build --release builds one program at -O2 in minutes and reuses the cached units. Build the base and the change in the same checkout, each with its own -o.

A database also needs its kernels. Without them every query runs on the interpreter. --warmup runs a warm-up against the server and links the kernels it uses:

sh
bin/gizlang build --release --warmup "bin/gizlang bench/db/engine/warm.giz 47610" db/pg/main.giz -o build/gizdb-ab \
    -- serve --port 47610 --bind 127.0.0.1 --dir build/warm-db

Before comparing, check that pg_stat_gizdb reports kernel_units_linked above 0 and no kernel_misses during the measured queries. Set GIZLANG_CLI to bin/gizlang while the server runs, so that a query shape the warm-up missed is compiled instead of interpreted.

Principles

Most past gains came from a few habits. The mechanisms are documented in Generated code and Hot paths.

  • Measure the owner. Fix the function with the largest cost first, and keep a change only when the number moves. An nsw hint that LLVM had already inferred changed nothing, and it was reverted.
  • Copy once. Parse in place, keep positions, and make an owned value only for what the caller receives.
  • Pay for the common case only. A fast path stays when its hit rate is measured and a test asserts the hit.
  • Size work to the input. A per-call table as large as the format's maximum made DEFLATE ten times slower on a 2 KB response.
  • Stay typed on hot paths. Writing a typed record field by field was 6.4 times faster than building and walking a generic JSON tree.
  • Share what threads compute. When the timeline shows the same function on many threads at once, compute it once.
  • Overlap waits with work. When the critical path shows a task waiting on the network while work that does not need the answer is ready, start the work first. GizBrowser loads a page's pictures while its scripts' timers run.
  • Keep memory in proportion to use. A cache of freed memory is bounded by what is in use now, not by the machine: malloc kept 64 MiB a size class on a 62 GB host, 100 MB of a page load's peak.
  • State the cost. A library function on a hot path documents what it allocates and copies, and the commit of an optimization includes its measurement.

Limits

  • It records on Linux only: each thread is sampled by a timer on its own CPU clock, which other systems lack. On macOS and Windows, run and start exit 6, and GIZLANG_PROFILE writes nothing.
  • The profile shows no kernel stacks and no GPU hardware counters: those need perf, eBPF, CUPTI or rocprofiler, which it does not use. capabilities lists each feature this host records, how exactly, and what it lacks.
  • It cannot attach to a process that it did not start (attach PID exits 6). Start the program with giz profile start (with --paused, to record only from attach on), or set GIZLANG_PROFILE (Settings).
  • Programs that are not Gizlang write no profile and have no stacks: the recorder reads their scheduler counters and wait channel each 10 ms (kin records), so their CPU, run-queue and waits are on the critical path, and the descendant a program waited for is followed. A child that ends within 10 ms has no look.
  • A thread's wait shorter than 10 ms counts as neither CPU nor wait; a task's waits are exact.
  • A process that is killed (SIGKILL, or a signal it does not handle) writes no end record, and the records of its last 10 ms are lost with its rings. quality names such processes and sets complete to false.
  • A thread is sampled from the first 1 ms of its life. A thread that lives less than 1 ms is CPU the samples do not cover.
  • Owners need Linux 5.17 or later (named anonymous memory). On an older kernel, all anonymous memory is malloc.
  • NUMA placement is the pages on each node, not which node each thread ran on. Memory bandwidth is not measured: it needs the uncore counters of perf.
  • Lock waits are exact for tasks (a channel, a join, a child or a thread they park on). A thread's wait in a futex (a C library lock) is the 10 ms wchan estimate.
  • For a GizOS guest, bin/gizlang tools/perf/names.giz --addresses gizos.elf names the addresses of a perf report.

The recorder is runtime/profile.giz. The wire numbers and fixed record sizes have one declaration, sdk/core/profile-format.giz, shared by the recorder, C hooks, SDK and reader. tools/perf/format.giz decodes it; metrics.giz owns selection and sample measurements; analysis.giz owns causal diagnosis and resource views; session.giz owns recording lifecycle and envelopes. display.giz renders answers without reading a recording. Source lines use the host's addr2line; symbol names use nm. capabilities reports whether source-line resolution through addr2line is available. Release packaging follows the reader's imports and preserves their relative paths under share/gizlang/profile, including the CLI command table. The packed reader runs without checkout sources. export --redact replaces recorded stack names with stable anonymous names in folded and pprof exports; weights and call structure remain available.