gizmodel/core/analyzer
Text analysis: the tokens of a text for the inverted index and for queries. Text is normalized (NFKC: full-width and compatibility forms become their plain ones) and case-folded (Unicode full folding); words are runs of letters and digits, and runs of Hangul, Han, Hiragana and Katakana become overlapping two-character grams (a one-character run stays one token), so Korean, Chinese and Japanese match inside words without a dictionary, as Lucene's CJK bigrams do. ASCII text takes a byte path. Segments store the tokens, not these rules: a change to the rules needs every text index rebuilt.
import * as Analyzer from "gizmodel/core/analyzer";Types
Word
type Word = { text: string; count: i64; prefix: bool; typos: i64; loose: bool; group: i64; letters: Vec<i64>; form: string; stemmed: string; }One word of a query: its term and how often the query repeats it. A typeahead word (loose) also reaches past its
exact term: the last word of open text as a prefix, typos edits in a word of 4 letters (1) or 8 (2), and in Hangul an
initial consonant for any syllable it starts (jamo initials U+314E U+3147 U+3139 match the corresponding syllable word, including while its final syllable is being composed). The words
of one group are readings of one thing typed: a row holds the group when it holds one of them, and the group scores
by its best term. letters are the code points of text, and form is the start that its other English forms share
(formPrefix): both are kept so a word compares with thousands of terms without taking them again.
text:stringcount:i64prefix:booltypos:i64loose:boolgroup:i64letters:Vec<i64>form:stringstemmed:string
Operators
type Operators = { scored: string; phrases: Vec<Vec<string>>; excluded: Vec<Vec<string>>; }The operators of a query text. "a b" is a phrase that every result holds, in that order. -word and -"a b" keep out
the rows that hold them. scored is the rest with the phrases' words: what ranks the rows and what a highlight marks.
A quote with no closing quote is plain words, so a phrase being typed still finds rows.
scored:stringphrases:Vec<Vec<string>>excluded:Vec<Vec<string>>
Reached
type Reached = { weight: f64; skip: i64; }What WORD makes of one term: its weight (Word.accepts), and skip, the byte length of a start of the term that no
term starting with it is within the typo limit of (0: none).
weight:f64skip:i64
Typing
type Typing = { q: Vec<i64>; limit: i64; prefix: bool; plain: bool; term: Vec<i64>; letters: Vec<i64>; cols: Vec<i64>; best: Vec<i64>; dead: i64; }The restricted Damerau-Levenshtein distance (a swap of neighbours is one edit) from a word's letters to terms, or to
a term's closest prefix when prefix, by columns of the term's letters: column j is the distance of each prefix of the
word to the term's first j letters (term, the letters of the term being measured). Sorted terms share prefixes, so letters keeps the last term's measured letters
and cols their columns (plain: the word has no Hangul), and the next term measures only past the letters it shares. A column past limit
everywhere stays past it (Ukkonen): dead is then its letter count, and no term with those first letters is within
the limit, unless a prefix already was (best).
q:Vec<i64>limit:i64prefix:boolplain:boolterm:Vec<i64>letters:Vec<i64>cols:Vec<i64>best:Vec<i64>dead:i64
Functions
tokens
function tokens(text: string): Vec<string>The tokens of text, in order (repeats kept: term frequency counts them).
Effects: fail, alloc.
words
function words(text: string, typeahead: bool): Vec<Word>The words of a query, in order, repeats counted, each distinct term a group. Typeahead reads each run of letters as one
group: its terms, and the run typed in the other Korean/English keyboard layout (dkssud matches its Hangul counterpart, and plan typed on a two-set layout matches plan),
since only one of the readings is what the person meant.
Effects: fail, alloc.
operators
function operators(text: string, typeahead: bool): OperatorsThe operators of TEXT. TYPEAHEAD: the text is still being typed, so its last word may grow.
Effects: fail, alloc.
groups
function groups(words: Vec<Word>): i64How many groups WORDS have.
Effects: fail.
markedExcerpt
function markedExcerpt(text: string, query: string, before: string, after: string, maxChars: i64, typeahead: bool = false): string throws stringA source-preserving excerpt of the words of QUERY (typeahead: what a typeahead MATCH reaches). The bound counts code points, including markers. Normalize grapheme clusters to map indexed terms back to the source. Merge overlapping CJK grams before marking, and never cut a grapheme cluster or an open marker pair.
Effects: fail, alloc.
sameForm
function sameForm(a: string, b: string): boolWhether A and B are forms of one English word: the same stem (stem). The index keeps the words as written, so a query reaches the other forms in each index's dictionary (inverted.giz resolve), and old indexes match them too.
Effects: fail, alloc.
formPrefix
function formPrefix(word: string): stringThe terms that may be forms of WORD start with this (stem with a final y dropped: studies for study).
Effects: fail, alloc.
stem
function stem(word: string): stringThe stem of an English word of lower-case ASCII letters, by its inflections only: a plural (-s, -es, -ies), a past (-ed, -ied) or a present participle (-ing), with a doubled final consonant undone and a final e dropped. A query and a term match when their stems are equal, so the rules need to agree with each other, not with a dictionary. Other words come back as they are.
Effects: fail, alloc.
Methods
Operators.admits
function Operators.admits(self: Operators, text: string): boolWhether a row whose text is TEXT holds every phrase of SELF and none of its excluded words.
Effects: fail, alloc.
Operators.admitsAny
function Operators.admitsAny(self: Operators, texts: Vec<string>): boolWhether a row whose columns hold TEXTS holds each phrase of SELF in one of them, and none of its excluded words in any. A phrase never runs from one column into the next.
Effects: fail, alloc.
Word.accepts
function Word.accepts(self: Word, term: string): f64The weight of TERM for WORD: 1 for its exact term, FORM for another English form of it, then NEAR, ONE_TYPO and TWO_TYPOS for a typeahead word; 0 when it does not reach it. One rule for the dictionary expansion of a query and for its highlight.
Effects: fail, alloc.
Word.reach
function Word.reach(self: Word, term: string, typing: mut Typing): ReachedWord.accepts with TYPING kept between the terms of a sorted dictionary, so a shared prefix is measured once and the
caller can jump past the terms that skip rules out (inverted.giz resolve).
Effects: fail, alloc.
Word.typing
function Word.typing(self: Word): TypingThe state of a fuzzy search for this word over sorted terms: no letter measured yet.
Effects: fail, alloc.