# gizmodel/core/analyzer

Text analysis: the tokens of a text for the inverted index and for queries. Text is normalized (NFKC: full-width and compatibility forms become their plain ones) and case-folded (Unicode full folding); words are runs of letters and digits, and runs of Hangul, Han, Hiragana and Katakana become overlapping two-character grams (a one-character run stays one token), so Korean, Chinese and Japanese match inside words without a dictionary, as Lucene's CJK bigrams do. ASCII text takes a byte path. Segments store the tokens, not these rules: a change to the rules needs every text index rebuilt.

```gizlang
import * as Analyzer from "gizmodel/core/analyzer";
```

## Types

### `Word`

```gizlang
type Word = { text: string; count: i64; prefix: bool; typos: i64; loose: bool; group: i64; letters: Vec<i64>; form: string; stemmed: string; }
```

One word of a query: its term and how often the query repeats it. A typeahead word (`loose`) also reaches past its
exact term: the last word of open text as a prefix, `typos` edits in a word of 4 letters (1) or 8 (2), and in Hangul an
initial consonant for any syllable it starts (jamo initials U+314E U+3147 U+3139 match the corresponding syllable word, including while its final syllable is being composed). The words
of one `group` are readings of one thing typed: a row holds the group when it holds one of them, and the group scores
by its best term. `letters` are the code points of `text`, and `form` is the start that its other English forms share
(formPrefix): both are kept so a word compares with thousands of terms without taking them again.

- `text`: `string`
- `count`: `i64`
- `prefix`: `bool`
- `typos`: `i64`
- `loose`: `bool`
- `group`: `i64`
- `letters`: `Vec<i64>`
- `form`: `string`
- `stemmed`: `string`

### `Operators`

```gizlang
type Operators = { scored: string; phrases: Vec<Vec<string>>; excluded: Vec<Vec<string>>; }
```

The operators of a query text. `"a b"` is a phrase that every result holds, in that order. `-word` and `-"a b"` keep out
the rows that hold them. `scored` is the rest with the phrases' words: what ranks the rows and what a highlight marks.
A quote with no closing quote is plain words, so a phrase being typed still finds rows.

- `scored`: `string`
- `phrases`: `Vec<Vec<string>>`
- `excluded`: `Vec<Vec<string>>`

### `Reached`

```gizlang
type Reached = { weight: f64; skip: i64; }
```

What WORD makes of one term: its weight (Word.accepts), and `skip`, the byte length of a start of the term that no
term starting with it is within the typo limit of (0: none).

- `weight`: `f64`
- `skip`: `i64`

### `Typing`

```gizlang
type Typing = { q: Vec<i64>; limit: i64; prefix: bool; plain: bool; term: Vec<i64>; letters: Vec<i64>; cols: Vec<i64>; best: Vec<i64>; dead: i64; }
```

The restricted Damerau-Levenshtein distance (a swap of neighbours is one edit) from a word's letters to terms, or to
a term's closest prefix when `prefix`, by columns of the term's letters: column j is the distance of each prefix of the
word to the term's first j letters (`term`, the letters of the term being measured). Sorted terms share prefixes, so `letters` keeps the last term's measured letters
and `cols` their columns (`plain`: the word has no Hangul), and the next term measures only past the letters it shares. A column past `limit`
everywhere stays past it (Ukkonen): `dead` is then its letter count, and no term with those first letters is within
the limit, unless a prefix already was (`best`).

- `q`: `Vec<i64>`
- `limit`: `i64`
- `prefix`: `bool`
- `plain`: `bool`
- `term`: `Vec<i64>`
- `letters`: `Vec<i64>`
- `cols`: `Vec<i64>`
- `best`: `Vec<i64>`
- `dead`: `i64`

## Functions

### `tokens`

```gizlang
function tokens(text: string): Vec<string>
```

The tokens of `text`, in order (repeats kept: term frequency counts them).

Effects: fail, alloc.

### `words`

```gizlang
function words(text: string, typeahead: bool): Vec<Word>
```

The words of a query, in order, repeats counted, each distinct term a group. Typeahead reads each run of letters as one
group: its terms, and the run typed in the other Korean/English keyboard layout (`dkssud` matches its Hangul counterpart, and `plan` typed on a two-set layout matches `plan`),
since only one of the readings is what the person meant.

Effects: fail, alloc.

### `operators`

```gizlang
function operators(text: string, typeahead: bool): Operators
```

The operators of TEXT. TYPEAHEAD: the text is still being typed, so its last word may grow.

Effects: fail, alloc.

### `groups`

```gizlang
function groups(words: Vec<Word>): i64
```

How many groups WORDS have.

Effects: fail.

### `markedExcerpt`

```gizlang
function markedExcerpt(text: string, query: string, before: string, after: string, maxChars: i64, typeahead: bool = false): string throws string
```

A source-preserving excerpt of the words of QUERY (typeahead: what a typeahead MATCH reaches). The bound counts code
points, including markers.
Normalize grapheme clusters to map indexed terms back to the source. Merge overlapping
CJK grams before marking, and never cut a grapheme cluster or an open marker pair.

Effects: fail, alloc.

### `sameForm`

```gizlang
function sameForm(a: string, b: string): bool
```

Whether A and B are forms of one English word: the same stem (stem). The index keeps the words as written, so a
query reaches the other forms in each index's dictionary (inverted.giz resolve), and old indexes match them too.

Effects: fail, alloc.

### `formPrefix`

```gizlang
function formPrefix(word: string): string
```

The terms that may be forms of WORD start with this (stem with a final y dropped: studies for study).

Effects: fail, alloc.

### `stem`

```gizlang
function stem(word: string): string
```

The stem of an English word of lower-case ASCII letters, by its inflections only: a plural (-s, -es, -ies), a past
(-ed, -ied) or a present participle (-ing), with a doubled final consonant undone and a final e dropped. A query and
a term match when their stems are equal, so the rules need to agree with each other, not with a dictionary. Other
words come back as they are.

Effects: fail, alloc.

## Methods

### `Operators.admits`

```gizlang
function Operators.admits(self: Operators, text: string): bool
```

Whether a row whose text is TEXT holds every phrase of SELF and none of its excluded words.

Effects: fail, alloc.

### `Operators.admitsAny`

```gizlang
function Operators.admitsAny(self: Operators, texts: Vec<string>): bool
```

Whether a row whose columns hold TEXTS holds each phrase of SELF in one of them, and none of its excluded words in any.
A phrase never runs from one column into the next.

Effects: fail, alloc.

### `Word.accepts`

```gizlang
function Word.accepts(self: Word, term: string): f64
```

The weight of TERM for WORD: 1 for its exact term, FORM for another English form of it, then NEAR, ONE_TYPO and
TWO_TYPOS for a typeahead word; 0 when it does not reach it. One rule for the dictionary expansion of a query and for
its highlight.

Effects: fail, alloc.

### `Word.reach`

```gizlang
function Word.reach(self: Word, term: string, typing: mut Typing): Reached
```

Word.accepts with TYPING kept between the terms of a sorted dictionary, so a shared prefix is measured once and the
caller can jump past the terms that `skip` rules out (inverted.giz resolve).

Effects: fail, alloc.

### `Word.typing`

```gizlang
function Word.typing(self: Word): Typing
```

The state of a fuzzy search for this word over sorted terms: no letter measured yet.

Effects: fail, alloc.

