# gizdb:core/analyzer

Text analysis: the tokens of a text for the inverted index and for queries. Text is normalized (NFKC: full-width and compatibility forms become their plain ones) and case-folded (Unicode full folding); words are runs of letters and digits, and runs of Hangul, Han, Hiragana and Katakana become overlapping two-character grams (a one-character run stays one token), so Korean, Chinese and Japanese match inside words without a dictionary, as Lucene's CJK bigrams do. ASCII text takes a byte path. Segments store the tokens, not these rules: a change to the rules needs every text index rebuilt.

```gizlang
import * as Analyzer from "gizdb:core/analyzer";
```

## Functions

### `tokens`

```gizlang
function tokens(text: string): Vec<string>
```

The tokens of `text`, in order (repeats kept: term frequency counts them).

Effects: fail.

### `markedExcerpt`

```gizlang
function markedExcerpt(text: string, query: string, before: string, after: string, maxChars: i64): Result<string, string>
```

A source-preserving excerpt. The bound counts code points, including markers.
Normalize grapheme clusters to map indexed terms back to the source. Merge overlapping
CJK grams before marking, and never cut a grapheme cluster or an open marker pair.

Effects: fail.

gizdb:core/analyzer

Text analysis: the tokens of a text for the inverted index and for queries. Text is normalized (NFKC: full-width and compatibility forms become their plain ones) and case-folded (Unicode full folding); words are runs of letters and digits, and runs of Hangul, Han, Hiragana and Katakana become overlapping two-character grams (a one-character run stays one token), so Korean, Chinese and Japanese match inside words without a dictionary, as Lucene's CJK bigrams do. ASCII text takes a byte path. Segments store the tokens, not these rules: a change to the rules needs every text index rebuilt.

gizlang

import * as Analyzer from "gizdb:core/analyzer";

Functions

`tokens`

gizlang

function tokens(text: string): Vec<string>

The tokens of text, in order (repeats kept: term frequency counts them).

Effects: fail.

`markedExcerpt`

gizlang

function markedExcerpt(text: string, query: string, before: string, after: string, maxChars: i64): Result<string, string>

A source-preserving excerpt. The bound counts code points, including markers. Normalize grapheme clusters to map indexed terms back to the source. Merge overlapping CJK grams before marking, and never cut a grapheme cluster or an open marker pair.

Effects: fail.