# gizlang:unicode ```gizlang import * as Unicode from "gizlang:unicode"; ``` ## Functions ### `codePoints` ```gizlang function codePoints(text: string): Vec ``` Generated from Unicode 17.0.0 UCD; do not edit tables. Unicode data: https://www.unicode.org/license.txt Unicode 17.0.0 algorithms. Tables are generated by generate.giz from the UCD. Lookup tables are bytes literals of fixed-width entries (Packed), read in place and searched in logarithmic time. Transformations allocate code points and one UTF-8 output; graphemes additionally copies each returned cluster. Canonical ordering is stable insertion ordering within a combining sequence (quadratic in a pathological descending sequence). Effects: fail. ### `fromCodePoints` ```gizlang function fromCodePoints(cps: Vec): string ``` The text made of these code points. Effects: fail. ### `fromUtf8Lossy` ```gizlang function fromUtf8Lossy(data: bytes): string ``` The text of UTF-8 bytes in which each ill-formed sequence is one U+FFFD, as Python's bytes.decode(errors="replace") and Hugging Face tokenizers decode it: a lead byte with fewer continuation bytes than it needs is one U+FFFD with those bytes; any other bad byte is one U+FFFD. Effects: fail. ### `utf16Units` ```gizlang function utf16Units(text: string): Vec ``` UTF-16 code units, including a surrogate pair for each supplementary scalar. Effects: fail. ### `fromUtf16Units` ```gizlang function fromUtf16Units(units: Vec): Option ``` Decode UTF-16. An unpaired surrogate has no host UTF-8 string value. Effects: fail. ### `toUpper` ```gizlang function toUpper(text: string): string ``` Full default case mapping; locale-specific Turkish/Lithuanian tailoring is absent. Effects: fail. ### `toLower` ```gizlang function toLower(text: string): string ``` The text with each letter in lower case. Effects: fail. ### `fold` ```gizlang function fold(text: string): string ``` Default full case folding (C/F entries, excluding Turkic T entries). Effects: fail. ### `normalize` ```gizlang function normalize(text: string, form: string): string requires form === "NFC" || form === "NFD" || form === "NFKC" || form === "NFKD" ``` TEXT in the Unicode normal form FORM: "NFC", "NFD", "NFKC" or "NFKD". Effects: fail. ### `graphemes` ```gizlang function graphemes(text: string): Vec ``` The extended grapheme clusters of the text, each as text. Effects: fail. ### `slug` ```gizlang function slug(text: string): string ``` A fragment id or address from `text`. It lowercases the NFC form, keeps each run of letters, marks and digits (General_Category L, M, N), and writes each run of other scalars as one "-", with no "-" at either end. It never fails; a text without letters or digits gives "". Effects: fail. ### `isEmoji` ```gizlang function isEmoji(grapheme: string): bool ``` True only for one RGI emoji from the pinned Unicode data. Effects: fail. ### `propertyRanges` ```gizlang function propertyRanges(key: string): Option> ``` The set of the property written as KEY in \p{KEY}: a binary property or a General_Category value by any of its names (`L`, `Letter`, `Alpha`), or NAME=VALUE with NAME one of General_Category, gc, Script, sc, Script_Extensions, scx. Names are matched exactly, as ECMAScript requires (no loose matching). Effects: fail. ### `hasProperty` ```gizlang function hasProperty(key: string, cp: i64): bool ``` Whether CP has the property written as KEY in \p{KEY} (as propertyRanges reads KEY); false for an unknown KEY. Effects: fail. ### `combiningClass` ```gizlang function combiningClass(cp: i64): i64 ``` The Canonical_Combining_Class of CP (9 is a virama). Effects: fail. ### `stringProperty` ```gizlang function stringProperty(name: string): Option>> ``` The sequences of the property of strings NAME (v flag), each a Vec of code points. RGI_Emoji is the union of the other six (UTS #51 ED-27), composed here so it is not stored twice. Effects: fail. ### `idStart` ```gizlang function idStart(cp: i64): bool ``` ID_Start, with the ECMAScript additions $ and _ excluded (the lexer adds them). Effects: fail. ### `idContinue` ```gizlang function idContinue(cp: i64): bool ``` Whether CP is a Unicode ID_Continue code point. Effects: fail. ### `spaceSeparator` ```gizlang function spaceSeparator(cp: i64): bool ``` Space_Separator (Zs), for the white space of the lexer and of trim. Effects: fail. ### `simpleFold` ```gizlang function simpleFold(cp: i64): i64 ``` scf: the simple case folding (CaseFolding C and S) of CP. Effects: fail. ### `simpleUpper` ```gizlang function simpleUpper(cp: i64): i64 ``` The simple uppercase mapping (UnicodeData field 12) of CP. Effects: fail. ### `foldOrbit` ```gizlang function foldOrbit(cp: i64): Vec ``` Every code point with the same simple case folding as CP, CP included, sorted. Effects: fail. ### `upperOrbit` ```gizlang function upperOrbit(cp: i64): Vec ``` Every code point with the same simple uppercase mapping as CP, CP included, sorted. Effects: fail. ### `simpleFoldPairs` ```gizlang function simpleFoldPairs(): Vec ``` The simple folding and simple uppercase tables as flat (code point, mapping) pairs, sorted, without identities. Effects: fail. ### `simpleUpperPairs` ```gizlang function simpleUpperPairs(): Vec ``` The simple uppercase mapping as flat (code point, mapping) pairs, sorted, without identities. Effects: fail.