# gizai/lora

the one reader of LoRA files. It reads every common layout (PEFT's and diffusers' lora_A and lora_B, kohya's lora_down and lora_up with an alpha, ComfyUI's diffusion_model prefix, kohya's names with underscores), scales each by alpha / rank and its strength, and stacks several files into one low-rank update a linear: A's rows and B's columns after each other, so N LoRAs cost one product of rank r1 + .. + rN. A model asks for a linear by its own name, as several of the file's linears when the model fuses them (`to_q`, `to_k`, `to_v` of one QKV: their B block-diagonal). A model merges the update into a weight (W + B A) or keeps it beside (Linear's a and b).

```gizlang
import * as Lora from "gizai/lora";
```

## Types

### `Spec`

```gizlang
type Spec = { path: string; strength: f64; }
```

A LoRA file and the strength it is applied at (1: as trained; 0 adds nothing; below 0 subtracts).

- `path`: `string`
- `strength`: `f64`

### `Pair`

```gizlang
type Pair = { down: Stored; up: Stored; scale: f64; }
```

A linear's update in one file: its down (A [rank, in]) and up (B [out, rank]) tensors and its scale, alpha / rank
times the file's strength.

- `down`: `gizlang_safetensors.Stored`
- `up`: `gizlang_safetensors.Stored`
- `scale`: `f64`

### `Lora`

```gizlang
type Lora = { files: Vec<Map<string, Pair>>; paths: Vec<string>; }
```

Several LoRA files read as one: each file's pairs by the model's name of their linear.

- `files`: `Vec<_gizlang_sdk_collections.Map<string, M21.Pair>>`
- `paths`: `Vec<string>`

### `Saved`

```gizlang
type Saved = { tensors: Map<string, Tensor>; state: Vec<TensorCore.Named>; }
```

A folder's adapter as `write` left it: its tensors by name and the run's state (named without `train.`).

- `tensors`: `_gizlang_sdk_collections.Map<string, _gizlang_lib_ai__tensor__storage.Tensor<f32>>`
- `state`: `Vec<M53.Named>`

## Functions

### `none`

```gizlang
function none(): Lora
```

No LoRA.

### `lora`

```gizlang
function lora(specs: Vec<Spec>): Lora throws string
```

The LoRAs `specs`, their tensors read when a model asks for them.

Effects: fail, read, write, console, io, alloc.

### `file`

```gizlang
function file(path: string): Lora throws string
```

The LoRA of file `path` at strength 1 ("": none).

Effects: fail, read, write, console, io, alloc.

### `specsOf`

```gizlang
function specsOf(j: Json): Vec<Spec> throws string
```

The LoRAs of a request (`loras`): a list of `{ path, strength }` (strength 1 when it has none) or of "PATH" or
"PATH:STRENGTH" texts, or a form field's one text of them apart by commas. A path is a .safetensors file or a PEFT
folder (adapter_config.json beside adapter_model.safetensors).

Effects: fail, io, alloc.

### `key`

```gizlang
function key(specs: Vec<Spec>): string
```

The name of LoRAs `specs` that a model keeps to know it has them: their paths and strengths.

Effects: fail, alloc.

### `start`

```gizlang
function start(key: TensorRandom.Key, inputs: i64, outputs: i64, rank: i64): Vec<Tensor>
```

A new pair of a linear of `inputs` to `outputs` at `rank`, as PEFT starts one: A [rank, inputs] uniform within
1 / sqrt(inputs) (kaiming-uniform), B [outputs, rank] zero, so the model starts as it was.

Effects: fail, alloc.

### `entries`

```gizlang
function entries(names: Vec<string>, widths: Vec<i64>, a: Tensor, b: Tensor): Vec<TensorCore.Named>
```

PEFT's entries of pair A ([rank, in]) and B ([out, rank]) of a linear made of the parts `names`, `widths` rows of B
each in order: each part takes A and its rows of B, as PEFT holds each module of a fused linear.

Effects: fail, read, write, console, io, alloc.

### `config`

```gizlang
function config(folder: string, names: Vec<string>, rank: i64, base: string, task: string): void throws string
```

Writes adapter_config.json of a LoRA of `rank` on model `base` for PEFT's `task`: its target modules are the last
parts of the linears' `names`.

Effects: fail, read, write, console, io, alloc.

### `write`

```gizlang
function write(folder: string, named: Vec<TensorCore.Named>, state: Vec<TensorCore.Named>, about: string = "{\"format\":\"pt\"}"): void throws string
```

Writes the adapter `named` (entries') in `folder` with a run's tensors `state` under `train.` and the metadata
`about` (a JSON object): one file, replaced whole.

Effects: fail, read, write, console, io, alloc.

### `saved`

```gizlang
function saved(folder: string): Option<Saved> throws string
```

The adapter in `folder`, none where it holds none.

Effects: fail, read, write, console, io, alloc.

## Methods

### `Lora.empty`

```gizlang
function Lora.empty(self: Lora): bool
```

Whether it holds no file.

### `Lora.pairsOf`

```gizlang
function Lora.pairsOf(self: Lora, name: string): Vec<Pair>
```

Every file's pair of the model's linear `name` (each A [rank, in], B [out, rank], its scale): a reader that merges
them itself (gizai/llm/convert's rows).

Effects: fail, alloc.

### `Lora.adapts`

```gizlang
function Lora.adapts(self: Lora, parts: Vec<string>): bool
```

Whether a file adapts any of the model's linears `parts`.

Effects: fail, alloc.

### `Lora.pair`

```gizlang
function Lora.pair(self: Lora, parts: Vec<string>, gpu: Option<Gpu>, outs: Vec<i64> = []): Vec<Tensor> throws string
```

The update of the model's linear made of the file's linears `parts` (their outputs after each other, as a fused
QKV's), where it runs (BF16 on a GPU, else F32): [A [R, in], B [out, R]], every file's pairs stacked, the scales in B;
[] when no file adapts it. Part j's rows take its B in its own columns, zeros elsewhere (block-diagonal). A file that
adapts some of the parts (PEFT's q_proj and v_proj of a fused QKV) leaves the others' `outs[j]` rows at zero; without
`outs` it is refused, as the rows of a part it lacks are not known.

Effects: fail, read, write, console, io, alloc.

### `Lora.merged`

```gizlang
function Lora.merged(self: Lora, w: Tensor, parts: Vec<string>): Tensor throws string
```

Weight `w` ([out, in]) of the model's linear made of the file's linears `parts`, plus their update B A, in w's dtype.

Effects: fail, read, write, console, io, alloc.

### `Lora.fits`

```gizlang
function Lora.fits(self: Lora, linears: Vec<string>): void throws string
```

Refuses the LoRA when a file adapts a linear that is not one of the model's `linears` (its names as Lora.pair takes
them): a LoRA of another model, applied in part, would draw what its trainer never saw.

Effects: fail, alloc.

### `Saved.of`

```gizlang
function Saved.of(self: Saved, folder: string, name: string): Tensor throws string
```

Tensor `name` of a saved adapter in `folder`.

Effects: fail, alloc.

## Constants

### `ADAPTER`

```gizlang
const ADAPTER = "adapter_model.safetensors"
```

The file of a PEFT folder that holds the adapter, and a training run's state under `train.`.

