GizStack
View .mdOpen in ChatGPTOpen in Claude

gizmodel/core/analyzer

Text analysis: the tokens of a text for the inverted index and for queries. Text is normalized (NFKC: full-width and compatibility forms become their plain ones) and case-folded (Unicode full folding); words are runs of letters and digits, and runs of Hangul, Han, Hiragana and Katakana become overlapping two-character grams (a one-character run stays one token), so Korean, Chinese and Japanese match inside words without a dictionary, as Lucene's CJK bigrams do. ASCII text takes a byte path. Segments store the tokens, not these rules: a change to the rules needs every text index rebuilt.

gizlang
import * as Analyzer from "gizmodel/core/analyzer";

Types

Word

gizlang
type Word = { text: string; count: i64; prefix: bool; typos: i64; loose: bool; group: i64; letters: Vec<i64>; form: string; stemmed: string; }

One word of a query: its term and how often the query repeats it. A typeahead word (loose) also reaches past its exact term: the last word of open text as a prefix, typos edits in a word of 4 letters (1) or 8 (2), and in Hangul an initial consonant for any syllable it starts (jamo initials U+314E U+3147 U+3139 match the corresponding syllable word, including while its final syllable is being composed). The words of one group are readings of one thing typed: a row holds the group when it holds one of them, and the group scores by its best term. letters are the code points of text, and form is the start that its other English forms share (formPrefix): both are kept so a word compares with thousands of terms without taking them again.

  • text: string
  • count: i64
  • prefix: bool
  • typos: i64
  • loose: bool
  • group: i64
  • letters: Vec<i64>
  • form: string
  • stemmed: string

Operators

gizlang
type Operators = { scored: string; phrases: Vec<Vec<string>>; excluded: Vec<Vec<string>>; }

The operators of a query text. "a b" is a phrase that every result holds, in that order. -word and -"a b" keep out the rows that hold them. scored is the rest with the phrases' words: what ranks the rows and what a highlight marks. A quote with no closing quote is plain words, so a phrase being typed still finds rows.

  • scored: string
  • phrases: Vec<Vec<string>>
  • excluded: Vec<Vec<string>>

Reached

gizlang
type Reached = { weight: f64; skip: i64; }

What WORD makes of one term: its weight (Word.accepts), and skip, the byte length of a start of the term that no term starting with it is within the typo limit of (0: none).

  • weight: f64
  • skip: i64

Typing

gizlang
type Typing = { q: Vec<i64>; limit: i64; prefix: bool; plain: bool; term: Vec<i64>; letters: Vec<i64>; cols: Vec<i64>; best: Vec<i64>; dead: i64; }

The restricted Damerau-Levenshtein distance (a swap of neighbours is one edit) from a word's letters to terms, or to a term's closest prefix when prefix, by columns of the term's letters: column j is the distance of each prefix of the word to the term's first j letters (term, the letters of the term being measured). Sorted terms share prefixes, so letters keeps the last term's measured letters and cols their columns (plain: the word has no Hangul), and the next term measures only past the letters it shares. A column past limit everywhere stays past it (Ukkonen): dead is then its letter count, and no term with those first letters is within the limit, unless a prefix already was (best).

  • q: Vec<i64>
  • limit: i64
  • prefix: bool
  • plain: bool
  • term: Vec<i64>
  • letters: Vec<i64>
  • cols: Vec<i64>
  • best: Vec<i64>
  • dead: i64

Functions

tokens

gizlang
function tokens(text: string): Vec<string>

The tokens of text, in order (repeats kept: term frequency counts them).

Effects: fail, alloc.

words

gizlang
function words(text: string, typeahead: bool): Vec<Word>

The words of a query, in order, repeats counted, each distinct term a group. Typeahead reads each run of letters as one group: its terms, and the run typed in the other Korean/English keyboard layout (dkssud matches its Hangul counterpart, and plan typed on a two-set layout matches plan), since only one of the readings is what the person meant.

Effects: fail, alloc.

operators

gizlang
function operators(text: string, typeahead: bool): Operators

The operators of TEXT. TYPEAHEAD: the text is still being typed, so its last word may grow.

Effects: fail, alloc.

groups

gizlang
function groups(words: Vec<Word>): i64

How many groups WORDS have.

Effects: fail.

markedExcerpt

gizlang
function markedExcerpt(text: string, query: string, before: string, after: string, maxChars: i64, typeahead: bool = false): string throws string

A source-preserving excerpt of the words of QUERY (typeahead: what a typeahead MATCH reaches). The bound counts code points, including markers. Normalize grapheme clusters to map indexed terms back to the source. Merge overlapping CJK grams before marking, and never cut a grapheme cluster or an open marker pair.

Effects: fail, alloc.

sameForm

gizlang
function sameForm(a: string, b: string): bool

Whether A and B are forms of one English word: the same stem (stem). The index keeps the words as written, so a query reaches the other forms in each index's dictionary (inverted.giz resolve), and old indexes match them too.

Effects: fail, alloc.

formPrefix

gizlang
function formPrefix(word: string): string

The terms that may be forms of WORD start with this (stem with a final y dropped: studies for study).

Effects: fail, alloc.

stem

gizlang
function stem(word: string): string

The stem of an English word of lower-case ASCII letters, by its inflections only: a plural (-s, -es, -ies), a past (-ed, -ied) or a present participle (-ing), with a doubled final consonant undone and a final e dropped. A query and a term match when their stems are equal, so the rules need to agree with each other, not with a dictionary. Other words come back as they are.

Effects: fail, alloc.

Methods

Operators.admits

gizlang
function Operators.admits(self: Operators, text: string): bool

Whether a row whose text is TEXT holds every phrase of SELF and none of its excluded words.

Effects: fail, alloc.

Operators.admitsAny

gizlang
function Operators.admitsAny(self: Operators, texts: Vec<string>): bool

Whether a row whose columns hold TEXTS holds each phrase of SELF in one of them, and none of its excluded words in any. A phrase never runs from one column into the next.

Effects: fail, alloc.

Word.accepts

gizlang
function Word.accepts(self: Word, term: string): f64

The weight of TERM for WORD: 1 for its exact term, FORM for another English form of it, then NEAR, ONE_TYPO and TWO_TYPOS for a typeahead word; 0 when it does not reach it. One rule for the dictionary expansion of a query and for its highlight.

Effects: fail, alloc.

Word.reach

gizlang
function Word.reach(self: Word, term: string, typing: mut Typing): Reached

Word.accepts with TYPING kept between the terms of a sorted dictionary, so a shared prefix is measured once and the caller can jump past the terms that skip rules out (inverted.giz resolve).

Effects: fail, alloc.

Word.typing

gizlang
function Word.typing(self: Word): Typing

The state of a fuzzy search for this word over sorted terms: no letter measured yet.

Effects: fail, alloc.