# gizlang:unicode

```gizlang
import * as Unicode from "gizlang:unicode";
```

## Functions

### `codePoints`

```gizlang
function codePoints(text: string): Vec<i64>
```

Generated from Unicode 17.0.0 UCD; do not edit tables.
Unicode data: https://www.unicode.org/license.txt
Unicode 17.0.0 algorithms. Tables are generated by generate.giz from the UCD.
Lookup tables are bytes literals of fixed-width entries (Packed), read in place and searched in logarithmic time.
Transformations allocate code points and one UTF-8 output; graphemes additionally
copies each returned cluster. Canonical ordering is stable insertion ordering
within a combining sequence (quadratic in a pathological descending sequence).

Effects: fail.

### `fromCodePoints`

```gizlang
function fromCodePoints(cps: Vec<i64>): string
```

The text made of these code points.

Effects: fail.

### `fromUtf8Lossy`

```gizlang
function fromUtf8Lossy(data: bytes): string
```

The text of UTF-8 bytes in which each ill-formed sequence is one U+FFFD, as Python's
bytes.decode(errors="replace") and Hugging Face tokenizers decode it: a lead byte with fewer continuation bytes
than it needs is one U+FFFD with those bytes; any other bad byte is one U+FFFD.

Effects: fail.

### `utf16Units`

```gizlang
function utf16Units(text: string): Vec<u16>
```

UTF-16 code units, including a surrogate pair for each supplementary scalar.

Effects: fail.

### `fromUtf16Units`

```gizlang
function fromUtf16Units(units: Vec<u16>): Option<string>
```

Decode UTF-16. An unpaired surrogate has no host UTF-8 string value.

Effects: fail.

### `toUpper`

```gizlang
function toUpper(text: string): string
```

Full default case mapping; locale-specific Turkish/Lithuanian tailoring is absent.

Effects: fail.

### `toLower`

```gizlang
function toLower(text: string): string
```

The text with each letter in lower case.

Effects: fail.

### `fold`

```gizlang
function fold(text: string): string
```

Default full case folding (C/F entries, excluding Turkic T entries).

Effects: fail.

### `normalize`

```gizlang
function normalize(text: string, form: string): string requires form === "NFC" || form === "NFD" || form === "NFKC" || form === "NFKD"
```

TEXT in the Unicode normal form FORM: "NFC", "NFD", "NFKC" or "NFKD".

Effects: fail.

### `graphemes`

```gizlang
function graphemes(text: string): Vec<string>
```

The extended grapheme clusters of the text, each as text.

Effects: fail.

### `slug`

```gizlang
function slug(text: string): string
```

A fragment id or address from `text`. It lowercases the NFC form, keeps each run of letters,
marks and digits (General_Category L, M, N), and writes each run of other scalars as one "-",
with no "-" at either end. It never fails; a text without letters or digits gives "".

Effects: fail.

### `isEmoji`

```gizlang
function isEmoji(grapheme: string): bool
```

True only for one RGI emoji from the pinned Unicode data.

Effects: fail.

### `propertyRanges`

```gizlang
function propertyRanges(key: string): Option<Vec<i64>>
```

The set of the property written as KEY in \p{KEY}: a binary property or a General_Category value by any of its
names (`L`, `Letter`, `Alpha`), or NAME=VALUE with NAME one of General_Category, gc, Script, sc, Script_Extensions, scx.
Names are matched exactly, as ECMAScript requires (no loose matching).

Effects: fail.

### `hasProperty`

```gizlang
function hasProperty(key: string, cp: i64): bool
```

Whether CP has the property written as KEY in \p{KEY} (as propertyRanges reads KEY); false for an unknown KEY.

Effects: fail.

### `combiningClass`

```gizlang
function combiningClass(cp: i64): i64
```

The Canonical_Combining_Class of CP (9 is a virama).

Effects: fail.

### `stringProperty`

```gizlang
function stringProperty(name: string): Option<Vec<Vec<i64>>>
```

The sequences of the property of strings NAME (v flag), each a Vec of code points. RGI_Emoji is the union of the
other six (UTS #51 ED-27), composed here so it is not stored twice.

Effects: fail.

### `idStart`

```gizlang
function idStart(cp: i64): bool
```

ID_Start, with the ECMAScript additions $ and _ excluded (the lexer adds them).

Effects: fail.

### `idContinue`

```gizlang
function idContinue(cp: i64): bool
```

Whether CP is a Unicode ID_Continue code point.

Effects: fail.

### `spaceSeparator`

```gizlang
function spaceSeparator(cp: i64): bool
```

Space_Separator (Zs), for the white space of the lexer and of trim.

Effects: fail.

### `simpleFold`

```gizlang
function simpleFold(cp: i64): i64
```

scf: the simple case folding (CaseFolding C and S) of CP.

Effects: fail.

### `simpleUpper`

```gizlang
function simpleUpper(cp: i64): i64
```

The simple uppercase mapping (UnicodeData field 12) of CP.

Effects: fail.

### `foldOrbit`

```gizlang
function foldOrbit(cp: i64): Vec<i64>
```

Every code point with the same simple case folding as CP, CP included, sorted.

Effects: fail.

### `upperOrbit`

```gizlang
function upperOrbit(cp: i64): Vec<i64>
```

Every code point with the same simple uppercase mapping as CP, CP included, sorted.

Effects: fail.

### `simpleFoldPairs`

```gizlang
function simpleFoldPairs(): Vec<i64>
```

The simple folding and simple uppercase tables as flat (code point, mapping) pairs, sorted, without identities.

Effects: fail.

### `simpleUpperPairs`

```gizlang
function simpleUpperPairs(): Vec<i64>
```

The simple uppercase mapping as flat (code point, mapping) pairs, sorted, without identities.

Effects: fail.

gizlang:unicode

gizlang

import * as Unicode from "gizlang:unicode";

Functions

`codePoints`

gizlang

function codePoints(text: string): Vec<i64>

Generated from Unicode 17.0.0 UCD; do not edit tables. Unicode data: https://www.unicode.org/license.txt Unicode 17.0.0 algorithms. Tables are generated by generate.giz from the UCD. Lookup tables are bytes literals of fixed-width entries (Packed), read in place and searched in logarithmic time. Transformations allocate code points and one UTF-8 output; graphemes additionally copies each returned cluster. Canonical ordering is stable insertion ordering within a combining sequence (quadratic in a pathological descending sequence).

Effects: fail.

`fromCodePoints`

gizlang

function fromCodePoints(cps: Vec<i64>): string

The text made of these code points.

Effects: fail.

`fromUtf8Lossy`

gizlang

function fromUtf8Lossy(data: bytes): string

The text of UTF-8 bytes in which each ill-formed sequence is one U+FFFD, as Python's bytes.decode(errors="replace") and Hugging Face tokenizers decode it: a lead byte with fewer continuation bytes than it needs is one U+FFFD with those bytes; any other bad byte is one U+FFFD.

Effects: fail.

`utf16Units`

gizlang

function utf16Units(text: string): Vec<u16>

UTF-16 code units, including a surrogate pair for each supplementary scalar.

Effects: fail.

`fromUtf16Units`

gizlang

function fromUtf16Units(units: Vec<u16>): Option<string>

Decode UTF-16. An unpaired surrogate has no host UTF-8 string value.

Effects: fail.

`toUpper`

gizlang

function toUpper(text: string): string

Full default case mapping; locale-specific Turkish/Lithuanian tailoring is absent.

Effects: fail.

`toLower`

gizlang

function toLower(text: string): string

The text with each letter in lower case.

Effects: fail.

`fold`

gizlang

function fold(text: string): string

Default full case folding (C/F entries, excluding Turkic T entries).

Effects: fail.

`normalize`

gizlang

function normalize(text: string, form: string): string requires form === "NFC" || form === "NFD" || form === "NFKC" || form === "NFKD"

TEXT in the Unicode normal form FORM: "NFC", "NFD", "NFKC" or "NFKD".

Effects: fail.

`graphemes`

gizlang

function graphemes(text: string): Vec<string>

The extended grapheme clusters of the text, each as text.

Effects: fail.

`slug`

gizlang

function slug(text: string): string

A fragment id or address from text. It lowercases the NFC form, keeps each run of letters, marks and digits (General_Category L, M, N), and writes each run of other scalars as one "-", with no "-" at either end. It never fails; a text without letters or digits gives "".

Effects: fail.

`isEmoji`

gizlang

function isEmoji(grapheme: string): bool

True only for one RGI emoji from the pinned Unicode data.

Effects: fail.

`propertyRanges`

gizlang

function propertyRanges(key: string): Option<Vec<i64>>

The set of the property written as KEY in \p{KEY}: a binary property or a General_Category value by any of its names (L, Letter, Alpha), or NAME=VALUE with NAME one of General_Category, gc, Script, sc, Script_Extensions, scx. Names are matched exactly, as ECMAScript requires (no loose matching).

Effects: fail.

`hasProperty`

gizlang

function hasProperty(key: string, cp: i64): bool

Whether CP has the property written as KEY in \p{KEY} (as propertyRanges reads KEY); false for an unknown KEY.

Effects: fail.

`combiningClass`

gizlang

function combiningClass(cp: i64): i64

The Canonical_Combining_Class of CP (9 is a virama).

Effects: fail.

`stringProperty`

gizlang

function stringProperty(name: string): Option<Vec<Vec<i64>>>

The sequences of the property of strings NAME (v flag), each a Vec of code points. RGI_Emoji is the union of the other six (UTS #51 ED-27), composed here so it is not stored twice.

Effects: fail.

`idStart`

gizlang

function idStart(cp: i64): bool

ID_Start, with the ECMAScript additions $ and _ excluded (the lexer adds them).

Effects: fail.

`idContinue`

gizlang

function idContinue(cp: i64): bool

Whether CP is a Unicode ID_Continue code point.

Effects: fail.

`spaceSeparator`

gizlang

function spaceSeparator(cp: i64): bool

Space_Separator (Zs), for the white space of the lexer and of trim.

Effects: fail.

`simpleFold`

gizlang

function simpleFold(cp: i64): i64

scf: the simple case folding (CaseFolding C and S) of CP.

Effects: fail.

`simpleUpper`

gizlang

function simpleUpper(cp: i64): i64

The simple uppercase mapping (UnicodeData field 12) of CP.

Effects: fail.

`foldOrbit`

gizlang

function foldOrbit(cp: i64): Vec<i64>

Every code point with the same simple case folding as CP, CP included, sorted.

Effects: fail.

`upperOrbit`

gizlang

function upperOrbit(cp: i64): Vec<i64>

Every code point with the same simple uppercase mapping as CP, CP included, sorted.

Effects: fail.

`simpleFoldPairs`

gizlang

function simpleFoldPairs(): Vec<i64>

The simple folding and simple uppercase tables as flat (code point, mapping) pairs, sorted, without identities.

Effects: fail.

`simpleUpperPairs`

gizlang

function simpleUpperPairs(): Vec<i64>

The simple uppercase mapping as flat (code point, mapping) pairs, sorted, without identities.

Effects: fail.