gizfs:bucket

An S3-compatible bucket of another provider (AWS S3, R2, MinIO, another GizFS gateway) as a Disk: the store keeps its chunk files there as it keeps them in a folder (diskOf), and nothing above the Disk knows which one it has. Each file is one object under PREFIX. A read open pins the object's ETag, so a later put fails its reads (STALE_RESOURCE) rather than mixing two versions; a write open holds the file in memory and puts it whole at sync or close, so a file written here is at most WRITE_LIMIT bytes (the store's chunk files are far smaller) and a bigger one goes through replaceRange, a multipart upload. A put is durable when it returns, so Disk.sync has nothing to do. Requests are signed with SigV4 (the canonical form is fs/s3.giz's, the one owner of it) and carry the SHA-256 of their body, so plain HTTP to a local endpoint is accepted too.

import * as Bucket from "gizfs:bucket";

Types

Bucket

type Bucket = { prefix: string; virtual: bool; bucket: string; host: string; calls: Task.Channel<Call>; }

The objects of BUCKET whose keys begin with PREFIX, as files named by the rest of the key, at HOST. VIRTUAL addresses the bucket as a host name (BUCKET.host, as AWS asks of new buckets), else as the first path segment.

• prefix: string
• virtual: bool
• bucket: string
• host: string
• calls: gizlang_task.Channel<M61.Call>

Object

type Object = { name: string; size: i64; etag: string; }

An object of a bucket: its name under the bucket's prefix, its size and its ETag, which changes when its bytes do.

• name: string
• size: i64
• etag: string

Functions

bucketDisk

function bucketDisk(sys: System, endpoint: string, region: string, bucket: string, prefix: string, access: string, secret: bytes, virtual: bool): Bucket throws string

The bucket BUCKET of the endpoint ENDPOINT (an http or https URL with no path) in REGION, signed with ACCESS and SECRET. Its requests go to a thread of its own (bucketThread), which signs them and keeps its connections: a Bucket holds no Ref, so a Disk that is one crosses Task.thread (Table.analyzeAside sends a table image there, its Disks with it) and works there. A Bucket that held its pool made every program that reaches both refuse to build (E_EFFECT, 10/27). Close ends the thread.

Effects: fail, read, write, console, io.

isBucket

function isBucket(disk: string): bool

True when DISK (an entry of node start --disks) names a bucket: an http(s) URL.

Effects: fail.

bucketAt

function bucketAt(sys: System, url: string): Bucket throws string

The bucket of URL, ENDPOINT/BUCKET[/PREFIX], with the credentials of the environment: GIZ_STORAGE_BUCKET_ACCESS and GIZ_STORAGE_BUCKET_SECRET, GIZ_STORAGE_BUCKET_REGION (us-east-1 when unset) and GIZ_STORAGE_BUCKET_VIRTUAL=1 for virtual-host addressing. The secret never goes on a command line.

Effects: fail, read, write, console, io.

Methods

Bucket.close

function Bucket.close(self: Bucket): void

Ends the Bucket's thread once its requests in flight are answered; later requests fail.

Effects: fail, io.

Bucket.objects

function Bucket.objects(self: Bucket, path: string, most: i64): Vec<Object> throws IoError

The objects under PATH ("" for all) at any depth, at most MOST, by name.

Effects: fail, io.

Bucket.entries

function Bucket.entries(self: Bucket, path: string): Entries throws IoError

The names under PATH ("" for all): one level, as a folder lists them.

Effects: fail, io.

Bucket.list

function Bucket.list(self: Bucket, path: string): Vec<string> throws IoError

The names under PATH, at most a million (more is an invalid argument).

Effects: fail, io.

Bucket.open

function Bucket.open(self: Bucket, path: string, mode: i64): File throws IoError

A read open pins the object's ETag; a write open (WRITE) holds the file in memory until its sync or close.

Effects: fail, io.

Bucket.keep

function Bucket.keep(self: Bucket, path: string, data: bytes): bool throws IoError

The put of a write open's held bytes: durable when it returns.

Effects: fail, io.

Bucket.readFile

function Bucket.readFile(self: Bucket, path: string): bytes throws IoError

The bytes of the object at PATH.

Effects: fail, io.

Bucket.replaceFile

function Bucket.replaceFile(self: Bucket, path: string, data: bytes): i64 throws IoError

Puts DATA as the object at PATH: durable when it returns.

Effects: fail, io.

Bucket.replaceRange

function Bucket.replaceRange(self: Bucket, path: string, source: File, start: i64, length: i64, windowBytes: i64): i64 throws IoError

LENGTH bytes of SOURCE from START as PATH, one part at a time: WINDOWBYTES does not apply.

Effects: fail, io.

Bucket.remove

function Bucket.remove(self: Bucket, path: string): bool throws IoError

False when PATH was not there (S3's DELETE answers 204 either way, so a HEAD asks first).

Effects: fail, io.

Bucket.link

function Bucket.link(self: Bucket, from: string, to: string): bool throws IoError

A copy: an object store names no object twice.

Effects: fail, read, write, console, io.

Bucket.sync

function Bucket.sync(self: Bucket): bool throws IoError

True: each put is durable when it returns, so nothing waits.

Constants

PART_BYTES

const PART_BYTES = 8388608

The bytes of one request body or ranged read: one multipart part, under gizlang:http's MOST_BODY.