# Tools All tools are Gizlang programs in `tools/`. `bin/giz` builds the toolchain (the compiler and the C runtime). It then runs the Gizlang command line, `tools/cli/main.giz`, for every command. A compiler edit starts from the build folder's last compiler, which is a fixed point of the previous sources. If that compiler emits the same IR, it is linked again. If not, the new IR is linked, and that compiler must compile its sources to the same IR (or link once more and do so). New syntax that the last compiler cannot compile, or no fixed point, starts from the seed (`bootstrap/compiler.ll.xz`). `giz bootstrap` always starts from the seed. On a shared server, a comment takes 6 s (the seed took 55 s). A code change compiles the one unit of each changed function and links in about 1.3 s (`ir_objects`, a unit per function; llvm-split's parts all missed the ThinLTO cache, 28-37 s). The tools and the prebuilt SDK are keyed by the compiler's IR (`compiler_ir`), so a comment change rebuilds neither. ## Contents 1. [Commands](#commands) 2. [REPL](#repl) 3. [Formatter](#formatter) 4. [C and TypeScript bindings](#c-and-typescript-bindings) 5. [JSON codecs](#json-codecs) 6. [Language server](#language-server) 7. [Debugger](#debugger) 8. [Private native release construction](#private-native-release-construction) ## Commands The command table in `tools/cli/commands.giz` is the one source of the commands. Each entry has the name, aliases, flags, operands, summary, examples, exit codes, the JSON output schema and the access (public, private or internal). These all come from it: - Dispatch. `gizlang` runs only a command in the table. An unknown command exits with status 2. - `giz help` lists the commands by group; `giz help --all` adds the commands that work on this repository. `giz help COMMAND` and `giz COMMAND --help` show one command. - `giz completion bash`. - `giz capabilities` (`gizlang.capabilities.v1`). - The `commands` of `giz doc --json`, and thus the command page of the site. `tests/tools/doc` runs each example of the table as a test. ### Reference `giz doc` assembles the reference of the SDK and the commands, as one versioned JSON (`gizlang.reference.v1`). The developer site, LSP hover, `llms.txt` and MCP read it. They do not parse sources. | Command | Result | | --- | --- | | `giz doc NAME` | The docs of a module (`gizlang:json`), a declaration (`Json.get`) or a command (`check`), as text. | | `giz doc --json` | The whole reference. | | `giz doc --json --package FILE` | The reference of one package (`package.json`) or file, not of the SDK. | | `giz doc --check` | Diagnostics `PATH:LINE: KIND: MESSAGE` for each problem, and exit status 1 if there is one. | | `--previous FILE --release VERSION` | `since` of each declaration from the reference of the previous release. New declarations get VERSION. The ids that went away are in `removed`. | The compiler owns the facts. The outline gives each public declaration, its types, its effects and its doc comment. `giz doc` adds: - The signature, from the declared source span, without the body, `export` or the native binding. A record with a hidden field (`_name`) shows only its name. A `_` name is not in the reference. - `refs`: each `[Name]` in a doc text, and each `@see` name, resolved to a declaration id (`gizlang:json#Json.get`). A link `[text](url)`, a code span, a fenced block and an index (`v[i]`) are not references. - The checks of `--check`: | Kind | Problem | | --- | --- | | `undocumented` | A public declaration has no doc comment. | | `reference` | A `[Name]` or `@see` name resolves to no public declaration. | | `example` | A Gizlang block of an `@example` is a whole program. It does not build, or exits with a status that is not 0. | `tests/tools/doc` holds the coverage ratchet of the SDK. Raise its numbers when you add docs. A release carries the reference as `share/gizlang/reference.json`, so an installed `giz doc` needs no SDK sources. Packages and registries are in [LANGUAGE](../reference/language.md#modules-and-packages). ## REPL `giz repl` reads, checks, runs and prints (`tools/repl/main.giz`). - Each entry is checked and run as part of one program. - Declarations (`import`, `function`, `type`, `interface`, `declare`) stay in the session. - Statements run in `main(sys: System)`, after the earlier statements of the session. The earlier statements run again with their output hidden, so their bindings stay visible. - The REPL prints the value of an expression. - An entry continues on the next line while its brackets and strings are open. - Commands: `:show` (the program of the session), `:reset` and `:quit`. ## Catalogs of messages `giz i18n DIR [--add TAG]... [--check]` brings the catalogs `locales/TAG.json` of the `.ghtml` files under DIR up to date: each new message is added with an empty translation, a key no file shows is removed, and each catalog prints how much of it is translated. `--check` writes nothing and fails on a catalog that is behind or a translation that does not fit ([MULTILINGUAL-ARCHITECTURE](localization.md)). ## Formatter `giz fmt [--check | --bytes] FILE...` lays out each line by its bracket depth: 4 spaces for each level, and one more level for a continued statement. It puts spaces between tokens by rule: - after `,` `;` `:`; - around binary operators and `=>`; - before `{`; - between keywords and `(`. It keeps the lines of the author, the comments, the inside of braces, and the spacing of `<` and `>`. Before the layout, `fmt` rewrites old spellings to the modern ones that the checker proves the same (`Option.some(x)` to `x`, `x.unwrap()` to `x!`, `join(v, s)` to `v.join(s)`, string `+` chains to template strings, and unused imports removed; [COMPILER](compiler.md#modernize)). `--check` does not rewrite. It changes only the files it is given; the files of one program share each round's one check. `--bytes` makes only one rewrite: a string's `s.length`, `s[i]`, `s.slice(a, b)`, `s.indexOf(x)` and `s.lastIndexOf(x)` become `s.byteLength`, `s.byteAt(i)`, `s.byteSlice(a, b)`, `s.byteIndexOf(x)` and `s.byteLastIndexOf(x)`, which always count UTF-8 bytes; it lays out nothing. A land needs no `fmt`: `wt.giz land` lays out the changed files that must stay laid out (`giz ci layout`); `fmt` is the owner's sweep of its paths. The formatter writes a file only when the lexer of the compiler (`--tokens`) finds the same tokens before and after. Thus it can format files that do not check yet, and it cannot change the meaning of a file. `--check` lists the files that are not formatted and exits with 1. ## C and TypeScript bindings ### C headers `giz bind HEADER LIBRARY NAME...` prints a `declare` line for each named C function of HEADER. It reads the JSON AST of clang through the shared owner `tools/bind/bindings.giz`. It resolves typedefs. It maps types as the table in [SYSTEM](system.md#native-functions) shows: | C | Gizlang | | --- | --- | | Integers and floats | By width. | | `const char *` | `string` | | `const` byte pointers | `bytes` | | Writable byte pointers | `mut bytes` | | Other writable scalar pointers | `mut` out-parameters | | `T **` | `mut u64` | | Any other pointer | `u64` | - A function that it cannot declare (variadic, a struct or function pointer by value, `long double`) becomes a comment that gives the reason. - clang shows an array parameter as a pointer. Thus a line with `mut` scalars ends with a comment that names them. Make sure that C writes one value there. - The output is source that you must review and paste. Its effects are the default `{io, fail}`. The manifest must still grant the library. Reader selection: - The repository reader uses `GIZLANG_CLANG` if it is set. If not, it uses `clang`. - An installed binding uses the admitted release backend, with the same target, relocated sysroot and Clang resource directory as native compilation. It ignores reader overrides from the environment. - A reader must exit successfully before its JSON is admitted. Valid JSON from a reader that failed is still an error. - The native entry `tools/bind/native.giz` imports the shared binding owner. Its import closure does not include the repository TypeScript adapter. Tests: - `tests/tools/bind/run.giz` binds zlib from its header, and checks it against the `crc32` of `gizlang:compress` and a round trip. - `tests/tools/bind/native/run.giz` specifies the selection of the native reader, the refusal of `.d.ts`, and source-disclosure admission. ### TypeScript declarations The repository adapter `tools/bind/main.giz` supports a `.d.ts` HEADER. It runs the checker of TypeScript on the VM in its own process (`tools/bind/typescript.js`, through `gizvm`). The `typescript` package is found from the folder of the file or from `NODE_PATH`. LIBRARY is the JavaScript module (`node:path`, `npm:slugify`). The output is a Gizlang module: for each name, a function that takes `sys: System` first, loads LIBRARY in the JavaScript realm and calls the export ([SYSTEM](system.md#javascript-and-python)). Each function `throws string`. - Of the overloads of a function, the adapter binds the first one whose parameters and result it can map: `string`, `number` (`f64`), `boolean` (`bool`), `void`, `string[]` (`Vec`) and `number[]` (`Vec`). Any other type is a gizvm `Value`. A `Promise` result is awaited. - Optional and rest parameters are left out, with a comment. - A `bigint` type makes the function a comment that gives the reason. - `tests/tools/bind/run.giz` binds `node:path` from `@types/node` and calls it. The native entry, which keeps its source private, returns `E_BIND_TYPESCRIPT_UNAVAILABLE` for `.d.ts`. It contains no TypeScript checker. Native TypeScript binding is not available. ## JSON codecs `giz codec FILE TYPE...` prints JSON functions for the named record types of FILE, and for the records that they contain (`tools/codec/main.giz`). - For each named type: `encodeT(value): string` and `decodeT(text): Result`. - For all types: `writeT` and `readT`, on the writer and pull reader of `gizlang:json`. Fields can be: - integers (`i8` to `i64`, `u8` to `u32`), `f64`, `bool` and `string`; - records of the file; - `Vec` or `Option` of these. Decode rules: - Integers are read exactly (also above 2^53) and checked against their width. - A missing field, a value of a different kind, and trailing text are decode errors, with their offset. - An `Option` field can be absent or `null`. - Unknown keys are skipped. Decoding builds each record with a literal that names every field. Thus, when a field is added or renamed, the old functions do not compile. Run the command again to replace them (`tests/tools/codec/run.giz`). ## Language server `giz lsp` runs the Gizlang language server (`tools/lsp/main.giz`) over stdin and stdout. - The source CLI builds the server and keeps it in the cache. An installed release uses its admitted prebuilt helper. - The protocol output contains only JSON-RPC framed with `Content-Length`. - The server binary also accepts the absolute path of the compiler binary as its first argument. - `System.exit(code)` ends the server without a print of a main result. It keeps the protocol framing and the exit status, and runs the cleanup of files and temporary directories. ### Supported protocol The server implements the [LSP 3.17 protocol](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/): - initialize, shutdown and exit; - full document sync; - diagnostics on open and change; - document symbols; - hover, with declaration signatures and inferred effects; - definition lookup and references; - completion: after `value.`, the fields and methods of its type ([COMPILER](compiler.md#completion)), also while the edit does not check; elsewhere, the declarations of the last outline and the keywords. - signature help: the parameters of the called function and the argument at the cursor, for a method from the compiler's completion of its receiver, for a function from the last outline. - formatting: the layout of `giz fmt`, as one edit of the whole buffer, and none when the layout would change a token; - rename of a declaration and its uses, by the compiler's rename transaction on a snapshot of the package ([COMPILER](compiler.md#rename)). The program must check after it, or the rename is refused. Each changed file is one edit of the whole file. `prepareRename` gives the name at the cursor. - semantic tokens: each name the outline binds in the file, typed by the declaration it names (`type`, `enum` for a union, `interface`, `function`, `method`, `variable` for a constant), with the modifier `declaration` at the declaration. Locals have none; the grammar colors them; - folding ranges of brackets and block comments over more than one line; - workspace symbols: the declarations of the programs of the open documents whose name contains the query, in any case. - code actions: the quick fix of each error whose diagnostic has a `fix` ([COMPILER](compiler.md#diagnostics-and-warnings)), such as `let` for a `const` that is assigned; - inlay hints: the type of each local whose type is not written, after its name ([COMPILER](compiler.md#hints)). A local of a generic function gets none, because its type differs by instance. Behavior: - Positions use UTF-16, also for astral text. - When a document closes, its diagnostics are cleared. Older document versions are ignored. - A file of a package is checked by one resident compiler of that package (`gizlang --serve`). Each change writes the package snapshot again and sends `edit FILE`, so a changed function body is checked alone. A single file is checked by a child process. Both use isolated temporary copies of the editor buffers and never change a source file. - The outline is made only for a symbol, hover, definition, reference or completion request, once for each valid buffer. - Headers have a limit of 8 KiB. Bodies have a limit of 16 MiB. - Incremental sync and range formatting are not advertised. ### Editor buffers and the checked closure - Full buffers become immutable overlays through the source snapshot owner of the CLI. - Relative imports keep their original layout. - A source below a package manifest uses its declared source closure. Root source pins can be stale in a working project. Restored dependencies must first agree with their original pins. - Manifest, import and native grants come from disk, never from editor buffers. Preview hashes exist only in the temporary snapshot. User sources and manifests are never written. - The compiler outline gives checked file spans and resolved declaration bindings ([the compiler internals](compiler.md#outline)). Definitions, hover and references use those bindings across the checked closure. An unresolved or unsupported binding returns no result. - Open buffers invalidate the affected contexts. Their checked closure can share one outline. - When a document closes, its overlay is discarded, and the remaining documents are read from disk again. - A maximum of 128 documents can be open. Their combined buffers must fit the source capacity of the compiler. - Document versions are signed LSP integers. Stale changes are ignored. - Only local `.giz` file URIs are admitted, with percent decoding and rejection of canonical-path collisions. - Checks stay synchronous. These operations are read-only, so they need no edit transaction. ### VS Code The extension is in `tools/vscode` of the repository, and each toolchain release carries it as `share/gizlang/vscode/gizlang.vsix`; `giz vscode` installs that file. It starts `giz lsp` and adds the Test Explorer, debugging ([Debugger](#debugger)), the `giz` task type and the Run, Debug, Check and Test File commands. To build it from a checkout: ```sh cd tools/vscode npm install npm run package code --install-extension gizlang-0.3.0.vsix ``` Set `gizlang.path` to the `giz` command when it is not on `PATH`. In a Gizlang checkout, set it to `/absolute/path/to/gizlang/bin/giz`, and set `GIZLANG_BUILD` in the environment of VS Code. ### Neovim For [Neovim 0.11+](https://neovim.io/doc/user/lsp.html), put this in `init.lua`. You do not need the lspconfig plugin. ```lua vim.filetype.add({ extension = { giz = 'gizlang' } }) vim.lsp.config('gizlang', { cmd = { '/absolute/path/to/gizlang/bin/giz', 'lsp' }, cmd_env = { GIZLANG_BUILD = '/absolute/build/directory' }, filetypes = { 'gizlang' }, root_markers = { 'package.json', '.git' }, }) vim.lsp.enable('gizlang') ``` ### Test Build the server before you run `bin/giz test tools/lsp`. The test starts that binary and drives a complete framed session through pipes. The test and the launcher inherit `GIZLANG_BUILD`. ## Debugger `giz build --debug FILE` builds the program at -O0 with DWARF for its Gizlang sources: line tables, a subprogram for each function, and the arguments and locals with their types (`tools/cli/debuginfo.giz`). gdb or lldb then stops at a `.giz` line (`break main.giz:12`) or a function (`break twice`), steps by source line, `bt` shows Gizlang names and lines, and `info locals` shows numbers, `bool`, records, and `string`, `bytes` and `Vec` as `{data, length}`. The VS Code extension (`tools/vscode/src/debug.ts`) builds the file with `giz build --debug` and runs it under `gdb -i=dap` (gdb 14 or later) or lldb-dap. gdb 15 starts the program when it receives `launch`, before VS Code sends the breakpoints, so the extension holds `launch` until `configurationDone`. `tests/tools/debug` checks the metadata and the binary's DWARF; the test hosts have no debugger. `tests/tools/vscode/manifest.test.giz` checks the debugger contribution. ## Private native release construction The source-checkout entry has a private construction adapter. Its source and specifications are implemented. ### Construct a candidate Run the adapter through the script command, with the normal source-launcher environment and a private request file: ```sh GIZLANG_BUILD=/absolute/private/build bin/giz tools/distribution/build.giz /absolute/private/request.json ``` The request uses the release metadata object of `tools/distribution/release.giz`. There is no other release schema. ```json { "operation": "construct", "source": "/absolute/source/gizlang", "output": "/absolute/private/candidate-payload", "metadataOutput": "/absolute/private/candidate.unsigned.json", "release": "INSERT_THE_EXISTING_RELEASE_METADATA_OBJECT_HERE", "inputs": { "libexec/llvm/clang": "/absolute/reviewed/clang" }, "probes": [ {"label": "reviewed private implementation", "fragment": "EXACT_REVIEWED_PRIVATE_SOURCE_FRAGMENT"} ] } ``` This excerpt shows only the placement of the fields. To make a real request: 1. Replace the `release` placeholder with the actual metadata object. 2. Put every prebuilt backend, runtime and sysroot input from its `files` in `inputs`. Each key is an exact payload path. Each value is an explicitly reviewed absolute source file. 3. Get the sizes and hashes from the release artifact audit. Never invent them. 4. Add reviewed probes of core implementation and paths. The Builder also keeps its mandatory probe of the known private TypeScript helper. Rules: - `release.files` contain only pinned prebuilt inputs. - Generated payload paths must not be in the input mapping. - Every source must agree with its `File` size and SHA-256 before construction accepts it. - The Builder owns the target layout, the generated SDK, compiler and tool artifacts, the source disclosure audit, compiler semantics and linking. - The request is a regular UTF-8 file, limited by `Release.MOST_METADATA`, and parsed with SDK `Json.parseUnique`. Input mapping, target and `File` admission stay the contracts of the Builder and Release owners. Environment and output: - `Common.environment(sys, true)` stays the one owner of the root, build, cache, compiler and CLI identities. - Construction requires `distribution.none`, an explicit source root that agrees, absolute tool, cache and build paths, and existing private output parents. - The output and the metadata must not exist, and must be physically outside the source checkout. The metadata also stays outside the payload. - The metadata file uses the atomic publication of the SDK that does not replace. Thus existing or concurrent outputs cannot be overwritten. - A failed build can leave its private candidate for inspection. Use new output paths. Do not use an incomplete candidate again. The adapter gives machine JSON with `signed:false` and `verified:false`. It writes only the unsigned canonical `Release.metadata` to `metadataOutput`. - The supplied evidence digest stays an input of the release identity. It is not a new proof that construction makes. - Signing, an installer bound to the signed envelope, actual target execution and verified public publication have their own owners. ### Construct an installer After the signed envelope exists, the same private script accepts `operation:"installer"`: ```sh GIZLANG_BUILD=/absolute/private/build bin/giz tools/distribution/build.giz /absolute/private/installer-request.json ``` ```json { "operation": "installer", "source": "/absolute/private/source-checkout", "output": "/absolute/private/gizlang-installer", "candidate": { "root": "/absolute/private/payload", "metadata": "/absolute/private/unsigned-release.json" }, "envelope": "/absolute/private/signed-envelope.json", "binding": { "target": "x86_64-unknown-linux-gnu", "product": "toolchain", "origin": "https://gizstack.com/", "signedMetadataSha256": "", "trusted": [{ "id": "", "publicKey": "<64 lowercase hex characters>" }], "httpsAuthority": "/absolute/reviewed/public-ca.pem" }, "probes": [{ "label": "reviewed private implementation", "fragment": "" }] } ``` The placeholders are not admissible on purpose. Supply the actual signed envelope digest, the public release keys and the reviewed implementation probes. - The candidate metadata is the unsigned `Release.metadata` of the earlier construction. It is not a second manifest. - The adapter reads the candidate metadata (maximum 2 MiB), the signed envelope bytes (maximum 4 MiB) and the public CA bytes (maximum 65,536 bytes) through `Dir.readBounded`. Then it gives admission and construction to their owners. - The output must not exist. It must be outside the source and the candidate, and separate from all request inputs. - Both operations use `Common.environment(sys, true)` and the same request parser and path admission. A successful installer output is machine JSON with `operation:"installer"`, `status:"constructed"`, `signed:false`, `verified:false`, and `installer:{source:,file:}`. - You can give its observed path, hash, size and executable pin directly to the private publication owner. Construction does not approve that publication. - It never signs, makes a catalog, changes download readiness, or certifies native or OS acceptance. - The flags refer to the installer artifact. The flags do not deny the authenticated release envelope that was an input. ### Installer trust `Builder.constructInstaller(sys, env, binding, candidate, signedMetadata, output, reviewedProbes)` authenticates the exact signed envelope through `Installer.admitted`. It compares the canonical `Release.metadata` identity with the constructed candidate. - `Binding.httpsAuthority` contains the reviewed public PEM CA bytes. They are embedded with the public release keys and the immutable envelope digest. - `Installer.httpsAuthoritySha256` identifies those exact bytes. `Release.trustedKeysSha256` identifies the canonical public key set. - Arguments at run time cannot change either trust source. - Metadata and every payload request use the same explicit HTTPS trust, independently of the certificate stores of the host. ### Installer link - Installer construction copies candidate bytes through `Native.auditTo` into private pinned snapshots. - It then uses the native link owner with the candidate backend, sysroot, startup, SDK and runtime, and the exact static OpenSSL and sqlite archives from `Common.sdkLibraryInputs`. - It does no host lookup of `-lssl`, `-lcrypto`, `-lsqlite3` or a separate `-luv`. - The final private installer output is published without replacement, through the atomic file owner. ### Specifications - `tests/tools/distribution/build.test.giz` specifies strict inputs, consistency of mapping and pins, explicit installer authority, key and path fields, private root separation, and refusal in an installed environment. - `tests/runtime/system/publication/run.giz` specifies atomic file conflicts, cleanup after a failed stream, unchanged ordinary replacement, and tree publication. - These specifications have not run yet.