# gizdb:search A search index of any source of documents: a folder (a POSIX tree or a GizFS mount), an S3 bucket, or a system of your own that implements Source. The index is a table of a GizDB database with BM25 indexes on each document's title (weight 3) and body, so a search finds as people type (typeahead MATCH: prefixes, typos, Hangul initials and the other keyboard layout), and Korean, Chinese and Japanese match inside words (db/core/analyzer.giz). The source is the truth: refresh lists it, reads again only the documents whose mark changed, and removes the ones that are gone, so a refresh after a change costs the change. The index holds nothing that the source cannot give again. Without a database, `memory` indexes a fixed set of documents in memory (a site's pages) with the same engine: BM25F over weighted fields, the operators and typeahead (db/core/inverted.giz best). ```gizlang import * as Search from "gizdb:search"; ``` ## Types ### `Listed` ```gizlang type Listed = { key: string; mark: string; title: string; meta: string; } ``` A document as its source lists it. `key` names it in the source; `mark` changes when its text does (a size and a modified time, an ETag, a version); `title` is searched with three times the weight of the body; `meta` is the caller's own text, kept as it is (the Drive keeps tags there). - `key`: `string` - `mark`: `string` - `title`: `string` - `meta`: `string` ### `Source` ```gizlang interface Source { documents(self: Self, most: i64): Vec throws string; text(self: Self, doc: Listed, most: i64): string throws string; } ``` A source of documents. - `documents` — The documents of the source, at most MOST. - `text` — The text of DOC, at most MOST bytes: "" when it has none (its title is still searched). ### `Refreshed` ```gizlang type Refreshed = { added: i64; changed: i64; removed: i64; } ``` What a refresh changed. - `added`: `i64` - `changed`: `i64` - `removed`: `i64` ### `Found` ```gizlang type Found = { key: string; title: string; meta: string; excerpt: string; score: f64; } ``` A document that holds the words: its listing, the text around the words, and its BM25 score. - `key`: `string` - `title`: `string` - `meta`: `string` - `excerpt`: `string` - `score`: `f64` ### `Folder` ```gizlang type Folder = { dir: Dir; } ``` The files of DIR at any depth, by path: a POSIX folder or a GizFS mount. A file's mark is its size and modified time; its title is its path as words. No dot entries. - `dir`: `gizlang_system.Dir` ### `Objects` ```gizlang type Objects = { bucket: Bucket; } ``` The objects of BUCKET at any depth (an AWS S3, R2 or MinIO bucket, or a GizFS gateway). An object's mark is its ETag; its title is its name as words. - `bucket`: `M61.Bucket` ### `Memory` ```gizlang type Memory = { fields: Vec; weights: Vec; texts: Vec>; } ``` Documents indexed in memory, without a database: row r is the r-th document, each the texts of its fields. - `fields`: `Vec` - `weights`: `Vec` - `texts`: `Vec>` ## Functions ### `prepare` ```gizlang function prepare(db: Database): bool throws StoreError ``` The index's table and its BM25 indexes in DB, made when they are not yet. Effects: fail, read, write, console, io. ### `refresh` ```gizlang function refresh(db: Database, source: Source): Refreshed throws StoreError ``` Brings the index in DB up to SOURCE: a document that is new or whose mark changed is read again, and one that the source no longer lists leaves. Effects: fail, read, write, console, io. ### `put` ```gizlang function put(db: Database, doc: Listed, text: string): bool throws StoreError ``` DOC with TEXT in the index at once, as a refresh would write it. Effects: fail, read, write, console, io. ### `find` ```gizlang function find(db: Database, words: string, under: string, most: i64, typeahead: bool, before: string, after: string): Vec throws StoreError ``` The documents whose key starts with UNDER ("" for all) that hold WORDS, best first: at most MOST. TYPEAHEAD finds as people type. BEFORE and AFTER mark the words in each excerpt. Effects: fail, read, write, console, io. ### `listed` ```gizlang function listed(db: Database, meta: string, most: i64): Vec throws StoreError ``` The listings of the documents whose meta matches the LIKE pattern META, by key: at most MOST. Effects: fail, read, write, console, io. ### `folder` ```gizlang function folder(dir: Dir): Source ``` DIR as a source of documents. ### `bucket` ```gizlang function bucket(b: Bucket): Source ``` The objects of B as a source of documents. ### `textOf` ```gizlang function textOf(name: string, data: bytes, most: i64): string ``` The text of a file named NAME that holds DATA, at most MOST bytes: a PDF's text, else the file itself when it is UTF-8 text, else "". Effects: fail, read, write, console, io. ### `memory` ```gizlang function memory(documents: Vec>, weights: Vec): Memory ``` The index of DOCUMENTS, each the texts of its fields in the order of WEIGHTS, the weight of each field (BM25F: a title of weight 3 counts three times a body of weight 1). Effects: fail. ## Methods ### `Memory.find` ```gizlang function Memory.find(self: Memory, words: string, most: i64, typeahead: bool): Vec ``` The rows that hold WORDS, best first: at most MOST, as MATCH finds them. TYPEAHEAD finds as people type. Effects: fail, read, write, console, io. ## Constants ### `MOST_TEXT` ```gizlang const MOST_TEXT = 1048576 ``` The most bytes of a document's text that the index keeps. ### `MOST_DOCUMENTS` ```gizlang const MOST_DOCUMENTS = 200000 ``` The most documents a refresh reads from its source.