# gizdb:search

A search index of any source of documents: a folder (a POSIX tree or a GizFS mount), an S3 bucket, or a system of your own that implements Source. The index is a table of a GizDB database with BM25 indexes on each document's title (weight 3) and body, so a search finds as people type (typeahead MATCH: prefixes, typos, Hangul initials and the other keyboard layout), and Korean, Chinese and Japanese match inside words (db/core/analyzer.giz). The source is the truth: refresh lists it, reads again only the documents whose mark changed, and removes the ones that are gone, so a refresh after a change costs the change. The index holds nothing that the source cannot give again.

Without a database, `memory` indexes a fixed set of documents in memory (a site's pages) with the same engine: BM25F over weighted fields, the operators and typeahead (db/core/inverted.giz best).

```gizlang
import * as Search from "gizdb:search";
```

## Types

### `Listed`

```gizlang
type Listed = { key: string; mark: string; title: string; meta: string; }
```

A document as its source lists it. `key` names it in the source; `mark` changes when its text does (a size and a
modified time, an ETag, a version); `title` is searched with three times the weight of the body; `meta` is the
caller's own text, kept as it is (the Drive keeps tags there).

- `key`: `string`
- `mark`: `string`
- `title`: `string`
- `meta`: `string`

### `Source`

```gizlang
interface Source { documents(self: Self, most: i64): Vec<Listed> throws string; text(self: Self, doc: Listed, most: i64): string throws string; }
```

A source of documents.

- `documents` — The documents of the source, at most MOST.
- `text` — The text of DOC, at most MOST bytes: "" when it has none (its title is still searched).

### `Refreshed`

```gizlang
type Refreshed = { added: i64; changed: i64; removed: i64; }
```

What a refresh changed.

- `added`: `i64`
- `changed`: `i64`
- `removed`: `i64`

### `Found`

```gizlang
type Found = { key: string; title: string; meta: string; excerpt: string; score: f64; }
```

A document that holds the words: its listing, the text around the words, and its BM25 score.

- `key`: `string`
- `title`: `string`
- `meta`: `string`
- `excerpt`: `string`
- `score`: `f64`

### `Folder`

```gizlang
type Folder = { dir: Dir; }
```

The files of DIR at any depth, by path: a POSIX folder or a GizFS mount. A file's mark is its size and modified
time; its title is its path as words. No dot entries.

- `dir`: `gizlang_system.Dir`

### `Objects`

```gizlang
type Objects = { bucket: Bucket; }
```

The objects of BUCKET at any depth (an AWS S3, R2 or MinIO bucket, or a GizFS gateway). An object's mark is its
ETag; its title is its name as words.

- `bucket`: `M61.Bucket`

### `Memory`

```gizlang
type Memory = { fields: Vec<TextIndex>; weights: Vec<f64>; texts: Vec<Vec<string>>; }
```

Documents indexed in memory, without a database: row r is the r-th document, each the texts of its fields.

- `fields`: `Vec<M93.TextIndex>`
- `weights`: `Vec<f64>`
- `texts`: `Vec<Vec<string>>`

## Functions

### `prepare`

```gizlang
function prepare(db: Database): bool throws StoreError
```

The index's table and its BM25 indexes in DB, made when they are not yet.

Effects: fail, read, write, console, io.

### `refresh`

```gizlang
function refresh(db: Database, source: Source): Refreshed throws StoreError
```

Brings the index in DB up to SOURCE: a document that is new or whose mark changed is read again, and one that the
source no longer lists leaves.

Effects: fail, read, write, console, io.

### `put`

```gizlang
function put(db: Database, doc: Listed, text: string): bool throws StoreError
```

DOC with TEXT in the index at once, as a refresh would write it.

Effects: fail, read, write, console, io.

### `find`

```gizlang
function find(db: Database, words: string, under: string, most: i64, typeahead: bool, before: string, after: string): Vec<Found> throws StoreError
```

The documents whose key starts with UNDER ("" for all) that hold WORDS, best first: at most MOST. TYPEAHEAD finds
as people type. BEFORE and AFTER mark the words in each excerpt.

Effects: fail, read, write, console, io.

### `listed`

```gizlang
function listed(db: Database, meta: string, most: i64): Vec<Listed> throws StoreError
```

The listings of the documents whose meta matches the LIKE pattern META, by key: at most MOST.

Effects: fail, read, write, console, io.

### `folder`

```gizlang
function folder(dir: Dir): Source
```

DIR as a source of documents.

### `bucket`

```gizlang
function bucket(b: Bucket): Source
```

The objects of B as a source of documents.

### `textOf`

```gizlang
function textOf(name: string, data: bytes, most: i64): string
```

The text of a file named NAME that holds DATA, at most MOST bytes: a PDF's text, else the file itself when it is
UTF-8 text, else "".

Effects: fail, read, write, console, io.

### `memory`

```gizlang
function memory(documents: Vec<Vec<string>>, weights: Vec<f64>): Memory
```

The index of DOCUMENTS, each the texts of its fields in the order of WEIGHTS, the weight of each field (BM25F: a
title of weight 3 counts three times a body of weight 1).

Effects: fail.

## Methods

### `Memory.find`

```gizlang
function Memory.find(self: Memory, words: string, most: i64, typeahead: bool): Vec<Hit>
```

The rows that hold WORDS, best first: at most MOST, as MATCH finds them. TYPEAHEAD finds as people type.

Effects: fail, read, write, console, io.

## Constants

### `MOST_TEXT`

```gizlang
const MOST_TEXT = 1048576
```

The most bytes of a document's text that the index keeps.

### `MOST_DOCUMENTS`

```gizlang
const MOST_DOCUMENTS = 200000
```

The most documents a refresh reads from its source.

gizdb:search

A search index of any source of documents: a folder (a POSIX tree or a GizFS mount), an S3 bucket, or a system of your own that implements Source. The index is a table of a GizDB database with BM25 indexes on each document's title (weight 3) and body, so a search finds as people type (typeahead MATCH: prefixes, typos, Hangul initials and the other keyboard layout), and Korean, Chinese and Japanese match inside words (db/core/analyzer.giz). The source is the truth: refresh lists it, reads again only the documents whose mark changed, and removes the ones that are gone, so a refresh after a change costs the change. The index holds nothing that the source cannot give again.

Without a database, memory indexes a fixed set of documents in memory (a site's pages) with the same engine: BM25F over weighted fields, the operators and typeahead (db/core/inverted.giz best).

gizlang

import * as Search from "gizdb:search";

Types

`Listed`

gizlang

type Listed = { key: string; mark: string; title: string; meta: string; }

A document as its source lists it. key names it in the source; mark changes when its text does (a size and a modified time, an ETag, a version); title is searched with three times the weight of the body; meta is the caller's own text, kept as it is (the Drive keeps tags there).

key: string
mark: string
title: string
meta: string

`Source`

gizlang

interface Source { documents(self: Self, most: i64): Vec<Listed> throws string; text(self: Self, doc: Listed, most: i64): string throws string; }

A source of documents.

documents — The documents of the source, at most MOST.
text — The text of DOC, at most MOST bytes: "" when it has none (its title is still searched).

`Refreshed`

gizlang

type Refreshed = { added: i64; changed: i64; removed: i64; }

What a refresh changed.

added: i64
changed: i64
removed: i64

`Found`

gizlang

type Found = { key: string; title: string; meta: string; excerpt: string; score: f64; }

A document that holds the words: its listing, the text around the words, and its BM25 score.

key: string
title: string
meta: string
excerpt: string
score: f64

`Folder`

gizlang

type Folder = { dir: Dir; }

The files of DIR at any depth, by path: a POSIX folder or a GizFS mount. A file's mark is its size and modified time; its title is its path as words. No dot entries.

dir: gizlang_system.Dir

`Objects`

gizlang

type Objects = { bucket: Bucket; }

The objects of BUCKET at any depth (an AWS S3, R2 or MinIO bucket, or a GizFS gateway). An object's mark is its ETag; its title is its name as words.

bucket: M61.Bucket

`Memory`

gizlang

type Memory = { fields: Vec<TextIndex>; weights: Vec<f64>; texts: Vec<Vec<string>>; }

Documents indexed in memory, without a database: row r is the r-th document, each the texts of its fields.

fields: Vec<M93.TextIndex>
weights: Vec<f64>
texts: Vec<Vec<string>>

Functions

`prepare`

gizlang

function prepare(db: Database): bool throws StoreError

The index's table and its BM25 indexes in DB, made when they are not yet.

Effects: fail, read, write, console, io.

`refresh`

gizlang

function refresh(db: Database, source: Source): Refreshed throws StoreError

Brings the index in DB up to SOURCE: a document that is new or whose mark changed is read again, and one that the source no longer lists leaves.

Effects: fail, read, write, console, io.

`put`

gizlang

function put(db: Database, doc: Listed, text: string): bool throws StoreError

DOC with TEXT in the index at once, as a refresh would write it.

Effects: fail, read, write, console, io.

`find`

gizlang

function find(db: Database, words: string, under: string, most: i64, typeahead: bool, before: string, after: string): Vec<Found> throws StoreError

The documents whose key starts with UNDER ("" for all) that hold WORDS, best first: at most MOST. TYPEAHEAD finds as people type. BEFORE and AFTER mark the words in each excerpt.

Effects: fail, read, write, console, io.

`listed`

gizlang

function listed(db: Database, meta: string, most: i64): Vec<Listed> throws StoreError

The listings of the documents whose meta matches the LIKE pattern META, by key: at most MOST.

Effects: fail, read, write, console, io.

`folder`

gizlang

function folder(dir: Dir): Source

DIR as a source of documents.

`bucket`

gizlang

function bucket(b: Bucket): Source

The objects of B as a source of documents.

`textOf`

gizlang

function textOf(name: string, data: bytes, most: i64): string

The text of a file named NAME that holds DATA, at most MOST bytes: a PDF's text, else the file itself when it is UTF-8 text, else "".

Effects: fail, read, write, console, io.

`memory`

gizlang

function memory(documents: Vec<Vec<string>>, weights: Vec<f64>): Memory

The index of DOCUMENTS, each the texts of its fields in the order of WEIGHTS, the weight of each field (BM25F: a title of weight 3 counts three times a body of weight 1).

Effects: fail.

Methods

`Memory.find`

gizlang

function Memory.find(self: Memory, words: string, most: i64, typeahead: bool): Vec<Hit>

The rows that hold WORDS, best first: at most MOST, as MATCH finds them. TYPEAHEAD finds as people type.

Effects: fail, read, write, console, io.

Constants

`MOST_TEXT`

gizlang

const MOST_TEXT = 1048576

The most bytes of a document's text that the index keeps.

`MOST_DOCUMENTS`

gizlang

const MOST_DOCUMENTS = 200000

The most documents a refresh reads from its source.