gizlang:text
Text and byte operations written in Gizlang over the built-in string primitives (slice, comparison, concatenation, push, string()). Strings are UTF-8: offsets count bytes, and a match always starts and ends on character boundaries because both sides are valid UTF-8.
import * as Text from "gizlang:text";Types
UnsignedParse
type UnsignedParse = | { kind: "ok"; value: u64; } | { kind: "invalid"; at: i64; } | { kind: "overflow"; at: i64; }Exact decimal unsigned parsing, retaining the first invalid/overflow byte.
okinvalidoverflow
IntegerParse
type IntegerParse = | { kind: "ok"; value: i64 } | { kind: "invalid"; offset: i64 } | { kind: "overflow"; offset: i64 }The result of reading an integer from text: the value, or where it is invalid or overflows.
okinvalidoverflow
VarintUnsigned
type VarintUnsigned = { value: u64; next: i64; }Bounded LEB128 offsets are byte positions, not characters or buffer slices.
value:u64next:i64
VarintSigned
type VarintSigned = { value: i64; next: i64; }A signed varint that was read: its value and the offset after it.
value:i64next:i64
VarintFailure
type VarintFailure = { message: string; next: i64; continuation: bool; }A varint that could not be read: why, the offset after it, and whether more bytes could finish it.
message:stringnext:i64continuation:bool
Decimal
type Decimal = { value: f64; end: i64; valid: bool; strict: bool; }The number at data[start..end): a sign, digits, a fraction and an exponent, each optional, read once into the nearest f64 and where it ends. valid: a digit before or after the point, and digits after an exponent mark (parseF64, gizlang:csv); strict, JSON's grammar: also digits on both sides of a point and no leading zero (gizlang:json).
value:f64end:i64valid:boolstrict:bool
Splitter
type Splitter = { size: i64; overlap: i64; keep: string; strip: bool; separators: Vec<string>; recursive: bool; }A splitter's settings: TextSplitter's defaults, then the keyword arguments. keep is "", "start" or "end".
size:i64overlap:i64keep:stringstrip:boolseparators:Vec<string>recursive:bool
Functions
parseU64
function parseU64(text: string): UnsignedParseTEXT as a decimal unsigned 64-bit integer, or where it is invalid or overflows.
Effects: fail.
appendCodePoint
function appendCodePoint(out: mut bytes, cp: i64): void requires cp> = 0 requires cp<SCALAR_LIMIT requires cp<SURROGATE_FIRST || cp>SURROGATE_LASTAppend one Unicode scalar as UTF-8, preserving the existing byte buffer.
Effects: fail.
uvarintSize
function uvarintSize(v: u64): i64The bytes that V takes as an unsigned LEB128 varint.
Effects: fail.
varintSize
function varintSize(v: i64): i64The bytes that V takes as a zigzag varint.
Effects: fail.
writeUVarint
function writeUVarint(out: mut bytes, v: u64): voidAppends V as an unsigned LEB128 varint.
Effects: fail.
writeVarint
function writeVarint(out: mut bytes, v: i64): voidSigned integers use zigzag; IEEE data and raw signed words have other encodings.
Effects: fail.
readUVarint
function readUVarint(data: bytes, offset: i64, bits: i64): Result<VarintUnsigned, string>bits is the declared integer width. Invalid/truncated/overflow input never faults or narrows a value silently. Existing nonminimal in-range LEB128 remains readable.
Effects: fail.
readUVarintWithin
function readUVarintWithin(data: bytes, offset: i64, end: i64, bits: i64): Result<VarintUnsigned, VarintFailure>The same decoder within an explicit byte range. Failures retain the first unread offset; continuation marks exhaustion of the declared word width.
Effects: fail.
readVarint
function readVarint(data: bytes, offset: i64, bits: i64): Result<VarintSigned, string>The zigzag varint at OFFSET of DATA and the offset after it. It fails when the varint is cut off or has more than BITS bits.
Effects: fail.
join
function join<T>(parts: Vec<T>, separator: string): stringThe items as text (${x}: numbers and bools too, as TypeScript joins them), SEPARATOR between each two.
frameStrings
function frameStrings(parts: Vec<string>): bytesAn unambiguous ordered string sequence: count, then each UTF-8 byte length and payload, all lengths unsigned 64-bit little endian. Delimiters in a payload cannot change boundaries. Suitable input to a domain-separated hash.
Effects: fail.
frameBytes
function frameBytes(parts: Vec<bytes>): bytesThe same framing engine for binary payloads. Callers admit output size.
Effects: fail.
unframeStrings
function unframeStrings(data: bytes, maxParts: i64, maxBytes: i64): Result<Vec<string>, string>Inverse of frameStrings; bounds are checked before allocating each payload.
Effects: fail.
unframeBytes
function unframeBytes(data: bytes, maxParts: i64, maxBytes: i64): Result<Vec<bytes>, string>The parts that frameBytes joined into DATA. It fails when there are more than MAXPARTS parts or a part has more than MAXBYTES bytes.
Effects: fail.
characters
function characters(text: string): i64The characters (code points) of a string.
Effects: fail.
characterList
function characterList(text: string): Vec<string>The characters (code points) of a string, each as a string; an incomplete sequence at the end is one.
Effects: fail.
grouped
function grouped(value: i64): stringvalue with thousands separators: 12,345.
Effects: fail.
clip
function clip(text: string, most: i64): stringTEXT cut to its first MOST characters (code points).
Effects: fail.
asciiSpace
function asciiSpace(byte: u64): boolWhether a byte is ASCII whitespace as C's isspace: space, tab, newline, vertical tab, form feed, return.
asciiDigit
function asciiDigit(byte: u64): boolWhether a byte is an ASCII decimal digit.
asciiAlpha
function asciiAlpha(byte: u64): boolAn ASCII letter.
asciiWord
function asciiWord(byte: u64): boolAn ASCII letter, digit or underscore: a byte of a word.
identifierByte
function identifierByte(c: u64): boolWhether byte C can be part of a JavaScript identifier: an ASCII letter or digit, _, $, or a byte of a non-ASCII character.
asciiPunct
function asciiPunct(byte: u64): boolAn ASCII punctuation mark: a printable byte that is not a letter, a digit or a space.
lowerAscii
function lowerAscii(byte: u64): u64One byte's ASCII lower case (headers, globs): other bytes unchanged.
Effects: fail.
sameFolded
function sameFolded(a: bytes, i: i64, b: bytes, j: i64, n: i64): boolWhether n bytes of a from i and of b from j are equal after ASCII lower-casing (header names, keywords).
Eight bytes are compared at a time; both ranges must lie in their data.
Effects: fail.
newlines
function newlines(data: bytes, from: i64, to: i64): i64The number of newline bytes in DATA from FROM up to TO.
Effects: fail.
lowerAsciiBytes
function lowerAsciiBytes(data: bytes): bytesASCII folding over arbitrary bytes; non-ASCII and invalid UTF-8 stay intact.
Effects: fail.
optionText
function optionText<T>(value: Option<T>): stringThe text of a TypeScript T | undefined: its value's, or "undefined".
parseInt
function parseInt(text: string, radix: i64 = 10): Option<i64>TypeScript's parseInt: after leading white space and a sign, the longest prefix of digits in RADIX (2 to 36; 16 also
after 0x, and a 0x prefix with no radix given), or none when there is no digit. It does not detect overflow.
Effects: fail.
parseFloat
function parseFloat(text: string): Option<f64>TypeScript's parseFloat: after leading white space, the longest prefix in JSON's number grammar (a leading + too), or none when there is none.
Effects: fail.
parseI64
function parseI64(text: string): IntegerParseA decimal integer with an optional leading minus sign.
Effects: fail.
writeU16LE
function writeU16LE(out: mut bytes, value: u64): voidAppends the low 2 bytes of VALUE, little-endian.
Effects: fail.
writeU16BE
function writeU16BE(out: mut bytes, value: u64): voidAppends the low 2 bytes of VALUE, big-endian.
Effects: fail.
writeU32LE
function writeU32LE(out: mut bytes, value: u64): voidAppends the low 4 bytes of VALUE, little-endian.
Effects: fail.
writeU32BE
function writeU32BE(out: mut bytes, value: u64): voidAppends the low 4 bytes of VALUE, big-endian.
Effects: fail.
writeU64LE
function writeU64LE(out: mut bytes, value: u64): voidAppends VALUE as 8 bytes, little-endian.
Effects: fail.
writeU64BE
function writeU64BE(out: mut bytes, value: u64): voidAppends VALUE as 8 bytes, big-endian.
Effects: fail.
writeF32LE
function writeF32LE(out: mut bytes, x: f64): voidAppends X as a 32-bit float, little-endian.
Effects: fail.
writeF32BE
function writeF32BE(out: mut bytes, x: f64): voidAppends X as a 32-bit float, big-endian.
Effects: fail.
writeF64LE
function writeF64LE(out: mut bytes, x: f64): voidAppends X as a 64-bit float, little-endian.
Effects: fail.
writeF64BE
function writeF64BE(out: mut bytes, x: f64): voidAppends X as a 64-bit float, big-endian.
Effects: fail.
signed
function signed(value: u64, bits: i64): i64 requires bits> = 1 requires bits<= 64 requires bits === 64 || (value>>u64(bits)) === 0u { const half = 1u<<u64(bits-1)The two's-complement value of the low bits bits, and back: signed(0xFFFFu, 16)
is -1 and unsigned(-1, 16) is 0xFFFF. Each requires its input to fit.
Effects: fail.
unsigned
function unsigned(value: i64, bits: i64): u64 requires bits> = 1 requires bits<= 64 requires value> = -i64((1u<<u64(bits-1))-1u)-1 requires value<= i64((1u<<u64(bits-1))-1u) { if(value> = 0) { return u64(value)The low BITS bits of VALUE in two's complement.
Effects: fail.
packI64s
function packI64s(out: mut bytes, v: Vec<i64>): voidLittle-endian arrays, 64 bytes a SIMD step: the same bytes as writeU64LE, writeF64LE, writeU32LE
or writeF32LE of each value, appended to out.
Effects: fail.
packF64s
function packF64s(out: mut bytes, v: Vec<f64>): voidAppends the values of V as 64-bit little-endian floats.
Effects: fail.
packU32s
function packU32s(out: mut bytes, v: Vec<u32>): voidAppends the values of V as 32-bit little-endian integers.
Effects: fail.
packF32s
function packF32s(out: mut bytes, v: Vec<f32>): voidAppends the values of V as 32-bit little-endian floats.
Effects: fail.
unpackI64s
function unpackI64s(data: bytes, at: i64, n: i64): Vec<i64>n values from data[at..], which must hold them.
Effects: fail.
unpackF64s
function unpackF64s(data: bytes, at: i64, n: i64): Vec<f64>N 64-bit little-endian floats from AT of DATA.
Effects: fail.
unpackU32s
function unpackU32s(data: bytes, at: i64, n: i64): Vec<u32>N 32-bit little-endian integers from AT of DATA.
Effects: fail.
unpackF32s
function unpackF32s(data: bytes, at: i64, n: i64): Vec<f32>N 32-bit little-endian floats from AT of DATA.
Effects: fail.
digitValue
function digitValue(byte: u64, base: i64): i64A byte's value as a digit of base (2-36: 0-9, then a-z or A-Z as
10-35), or -1: one rule for parseRadix and every hex reader (gizlang:json,
gizweb, gizlang:http).
Effects: fail.
parseRadix
function parseRadix(text: string, base: i64): IntegerParse requires base> = 2 requires base<= 36A non-negative integer in base 2..36, digits in either case (the inverse of radix).
Effects: fail.
radix
function radix(value: u64, base: i64): string requires base> = 2 requires base<= 36value in base 2..36 with lowercase digits, as JavaScript's toString(base).
Effects: fail.
fixed
function fixed(x: f64, digits: i64): string requires digits> = 0 requires digits<= 64x with digits digits after the point, rounded half to even from x's exact
binary value, as C's printf("%.Nf") and Rust's format!("{:.N}"): checksums
and reports print the same text in every language. Pure Gizlang over a
little big integer (32-bit limbs): the mantissa is found by scaling by 2,
which is exact, so no bit cast is needed.
Effects: fail.
formatF64
function formatF64(x: f64): stringx as string(x) prints it, which the compiler lowers to this: writeF64's text.
Effects: fail.
writeF64
function writeF64(out: mut bytes, x: f64): voidx as JavaScript's String(x) writes it, appended to out: the shortest digits that read back as x (Schubfach, as Java's Double.toString: one 126-bit multiplication by a power of ten from a table, no loop over precisions), placed as 123.5, 1e+21, 0.000001 or 1e-7; NaN, Infinity, and 0 for -0.
Effects: fail.
base64Encode
function base64Encode(data: bytes): stringDATA in base64 (RFC 4648), with padding.
Effects: fail.
base64UrlEncode
function base64UrlEncode(data: bytes): stringRFC 4648 URL alphabet, without padding.
Effects: fail.
base64UrlDecode
function base64UrlDecode(text: string): Option<bytes>Strict unpadded RFC 4648 URL text. Reject a noncanonical last sextet.
Effects: fail.
hexEncode
function hexEncode(data: bytes): stringTwo lowercase hexadecimal digits per byte (digests, tokens, wire hashes): 16 bytes a step, each nibble one table lookup (pshufb or NEON tbl) and the digits interleaved by shuffles; byte by byte below that.
Effects: fail.
hexDecode
function hexDecode(text: string): Option<bytes>Whole bytes in either hexadecimal case; reject odd widths, whitespace and non-hex. Validate before allocation so malformed input never leaves a partial decoded buffer.
Effects: fail.
hexDecodePrefix
function hexDecodePrefix(data: bytes): bytesHexadecimal pairs of DATA up to the first pair with a non-hex digit or a lone last digit, as Node's Buffer.from(s, 'hex') forgives. hexDecode is the strict one.
Effects: fail.
base64DecodeLenient
function base64DecodeLenient(data: bytes): bytesBase64 of DATA in either alphabet, up to the first '=', skipping any other character, as Node's Buffer.from(s, 'base64') forgives. base64Decode is the strict one.
Effects: fail.
base64Decode
function base64Decode(text: string): Option<bytes>The bytes of padded Base64 text, or none for any other character, a bad length or bad padding.
Effects: fail.
decimalPrefix
function decimalPrefix(data: bytes, start: i64, end: i64): Decimal requires start> = 0 requires start<= end requires end<= len(data) { let i = startThe decimal number in DATA from START to END, read into the nearest f64, and where it ends.
Effects: fail.
quoteIdentifier
function quoteIdentifier(name: string): stringOne SQL delimited identifier, preserving every character and doubling embedded quotes. Always quote: this is independent of a server's keyword set and needs no network client.
Effects: fail.
quoteIdentifierPath
function quoteIdentifierPath(name: string): stringA dotted name with each part quoted: schema.table is "schema"."table".
Effects: fail.
quoteLiteral
function quoteLiteral(text: string): stringOne single-quoted literal (SQL, YAML): the text between single quotes, each embedded quote doubled.
Effects: fail.
quoteShell
function quoteShell(text: string): stringOne word for a POSIX shell: the text between single quotes, each embedded quote closed, escaped and reopened.
Effects: fail.
shellCommand
function shellCommand(argv: Vec<string>): stringA command line for a POSIX shell: each word of ARGV quoted, joined by spaces.
Effects: fail.
parseF64
function parseF64(text: string): Option<f64>A whole string in JSON's number grammar, also with a leading + or zeros, as the nearest f64; none for anything else.
Effects: fail.
splitter
function splitter(kind: string): Splitter throws stringThe splitter of kind with TextSplitter's defaults: character, recursive_character, python, markdown or latex.
Effects: fail.
Methods
IntegerParse.valueOr
function IntegerParse.valueOr(self: IntegerParse, fallback: i64): i64The parsed value, or fallback when the text is not an integer or overflows.
IntegerParse.value
function IntegerParse.value(self: IntegerParse): i64The parsed value; a text that is not an integer is a broken invariant (assert).
Effects: fail.
bytes.isUtf8
function bytes.isUtf8(self: bytes, start: i64, end: i64): bool requires start> = 0 requires start<= end requires end<= len(self) { return utf8Valid(self, start, end)Whether bytes start..end are valid UTF-8, so string() of them cannot fail.
Effects: fail.
bytes.indexOf
function bytes.indexOf(self: bytes, needle: bytes, start: i64): i64 requires start> = 0 requires start<= len(self)The first byte offset of needle at or after start, or -1: the offsets whose first and last bytes match (W. Muła), 128 a step while none does, each then compared; one byte by one compare a block.
Effects: fail.
bytes.indexOfAny
function bytes.indexOfAny(self: bytes, set: bytes, start: i64): i64 requires start> = 0 requires start<= len(self)The first byte offset at or after start of any byte in set, or -1: up to four members by comparison, more by a 256-bit map read with two nibble lookups (each lane's row for its low nibble, and its high nibble's bit).
Effects: fail.
bytes.positions
function bytes.positions(self: bytes, start: i64, end: i64, set: bytes, out: mut Vec<i64>): i64 requires start> = 0 requires start<= end requires end<= len(self) { return self.walkSet(start, end, set, out, true)The structural index of self[start, end): the offset of every byte that is in set, in order, appended to out (simdjson's stage 1 without its string mask). A parser of delimited records (COPY text and CSV, JSON, HTTP headers) then walks the hits, not the bytes: 64 bytes a step, up to four members by comparison, more by the nibble lookups of indexOfAny. Returns the count appended.
Effects: fail.
bytes.countAny
function bytes.countAny(self: bytes, start: i64, end: i64, set: bytes): i64 requires start> = 0 requires start<= end requires end<= len(self) { let none: Vec<i64> = []The count of bytes of self[start, end) that are in set (bytes.positions without its index).
Effects: fail.
string.byteLength
function string.byteLength(self: string): i64The UTF-8 bytes of the text.
string.byteAt
function string.byteAt(self: string, index: i64): u64Byte INDEX of the text's UTF-8 (R001 out of range).
Effects: fail.
string.byteSlice
function string.byteSlice(self: string, start: i64, end: i64): stringBytes START to END of the text (R001 for a bad range or a split UTF-8 character).
Effects: fail.
string.byteIndexOf
function string.byteIndexOf(self: string, needle: string, from: i64 = 0): i64The first byte offset of NEEDLE at or after byte FROM (clamped to the text), or -1.
Effects: fail.
string.indexOf
function string.indexOf(self: string, needle: string, from: i64 = 0): i64The first place of NEEDLE at or after FROM (clamped to the text, as TypeScript does), or -1.
Effects: fail.
string.includes
function string.includes(self: string, needle: string): boolWhether NEEDLE occurs in the string.
Effects: fail.
string.startsWith
function string.startsWith(self: string, prefix: string, position: i64 = 0): boolWhether prefix is at byte POSITION (0 by default; clamped to the text, as in JavaScript), bytewise, without a copy: a slice could split a UTF-8 character (R001).
Effects: fail.
bytes.startsWith
function bytes.startsWith(self: bytes, prefix: bytes, position: i64 = 0): boolWhether prefix is at POSITION (0 by default; clamped to self, as in JavaScript), without a copy.
Effects: fail.
string.endsWith
function string.endsWith(self: string, suffix: string): boolWhether the string ends with SUFFIX.
Effects: fail.
string.split
function string.split(self: string, separator: string): Vec<string> requires separator.byteLength>0 { const text = utf8(self)The pieces between separators; an empty separator is not allowed.
Effects: fail.
string.replaceAll
function string.replaceAll(self: string, from: string, to: string): string requires from.byteLength>0 { return join(self.split(from), to)The string with each FROM replaced by TO.
Effects: fail.
string.repeat
function string.repeat(self: string, count: i64): string requires count> = 0 { let text = ""The string COUNT times.
Effects: fail.
string.trim
function string.trim(self: string): stringWithout leading and trailing ASCII whitespace.
Effects: fail.
string.trimStart
function string.trimStart(self: string): stringThe string without the ASCII whitespace at its start.
Effects: fail.
string.trimEnd
function string.trimEnd(self: string): stringThe string without the ASCII whitespace at its end.
Effects: fail.
string.lastIndexOf
function string.lastIndexOf(self: string, needle: string): i64The last byte offset of needle, or -1; "" is found at the end. Bytewise, as startsWith: a slice at each offset would split a UTF-8 character (R001).
Effects: fail.
string.byteLastIndexOf
function string.byteLastIndexOf(self: string, needle: string): i64The last byte offset of NEEDLE, or -1; "" is found at the end.
Effects: fail.
string.padStart
function string.padStart(self: string, length: i64, fill: string): stringPadded with fill (repeated, the last copy cut) to length bytes; a longer string is unchanged.
Effects: fail.
string.padEnd
function string.padEnd(self: string, length: i64, fill: string): stringThe string followed by copies of FILL up to LENGTH bytes.
Effects: fail.
string.toUpperCase
function string.toUpperCase(self: string): stringASCII case mapping; other characters are unchanged.
Effects: fail.
string.toLowerCase
function string.toLowerCase(self: string): stringThe string with A to Z in lower case (ASCII only).
Effects: fail.
string.localeCompare
function string.localeCompare(self: string, other: string): i64-1, 0 or 1 as SELF comes before, with or after OTHER in code point order (UTF-8 byte order): TypeScript's
localeCompare without a locale's collation, so xs.toSorted((a, b) => a.localeCompare(b)) sorts.
Effects: fail.
string.charAt
function string.charAt(self: string, index: i64): stringThe UTF-16 unit at INDEX as a string, or "" out of range.
Effects: fail.
string.substring
function string.substring(self: string, start: i64, end?: i64): stringTypeScript's substring: the text from START to END (the length when left out), each clamped to 0..length, in either order.
Effects: fail.
f64.toFixed
function f64.toFixed(self: f64, digits: i64): stringThe value with DIGITS digits after the point (0 to 20), rounded half away from zero: TypeScript's toFixed for values under 1e21.
Effects: fail.
i64.toFixed
function i64.toFixed(self: i64, digits: i64): stringThe integer with DIGITS zeros after the point.
Effects: fail.
bytes.readU32LE
function bytes.readU32LE(self: bytes, offset: i64): u64 requires offset> = 0 requires offset<= len(self)-4The unsigned 32-bit little-endian integer at OFFSET.
Effects: fail.
bytes.readU64LE
function bytes.readU64LE(self: bytes, offset: i64): u64One bounds check and one load; out of range faults as indexing does.
Effects: fail.
bytes.copyFrom
function bytes.copyFrom(self: mut bytes, at: i64, source: bytes, from: i64, size: i64): voidWrites SIZE bytes of SOURCE from FROM over this buffer from AT, in place, as one copy: a loop of self[i] = x through
a field path made the buffer its own again at each byte. A range outside its buffer faults as indexing does.
Effects: fail.
bytes.copyWithin
function bytes.copyWithin(self: mut bytes, at: i64, from: i64, size: i64): voidCopies SIZE bytes of this buffer from FROM to AT, in place; the ranges may overlap (bytes.copyFrom).
Effects: fail.
bytes.readU32BE
function bytes.readU32BE(self: bytes, offset: i64): u64 requires offset> = 0 requires offset<= len(self)-4The unsigned 32-bit big-endian integer at OFFSET.
Effects: fail.
bytes.readU16LE
function bytes.readU16LE(self: bytes, offset: i64): u64The unsigned 16-bit little-endian integer at OFFSET.
Effects: fail.
bytes.readU16BE
function bytes.readU16BE(self: bytes, offset: i64): u64The unsigned 16-bit big-endian integer at OFFSET.
Effects: fail.
bytes.readU64BE
function bytes.readU64BE(self: bytes, offset: i64): u64The unsigned 64-bit big-endian integer at OFFSET.
Effects: fail.
bytes.readF32LE
function bytes.readF32LE(self: bytes, offset: i64): f64The 32-bit little-endian float at OFFSET.
Effects: fail.
bytes.readF32BE
function bytes.readF32BE(self: bytes, offset: i64): f64The 32-bit big-endian float at OFFSET.
Effects: fail.
bytes.readF64LE
function bytes.readF64LE(self: bytes, offset: i64): f64The 64-bit little-endian float at OFFSET.
Effects: fail.
bytes.readF64BE
function bytes.readF64BE(self: bytes, offset: i64): f64The 64-bit big-endian float at OFFSET.
Effects: fail.
bytes.setU16LE
function bytes.setU16LE(self: mut bytes, offset: i64, value: u64): voidWrites the low 2, 4 or 8 bytes of value at offset, little-endian, over the bytes there.
Effects: fail.
bytes.setU32LE
function bytes.setU32LE(self: mut bytes, offset: i64, value: u64): voidWrites the low 4 bytes of VALUE at OFFSET, little-endian.
Effects: fail.
bytes.setU64LE
function bytes.setU64LE(self: mut bytes, offset: i64, value: u64): voidWrites VALUE at OFFSET as 8 bytes, little-endian.
Effects: fail.
Splitter.split
function Splitter.split(self: Splitter, text: string): Vec<string>The chunks of text: split_text of the splitter.
Effects: fail.
Constants
SCALAR_LIMIT
const SCALAR_LIMIT = 1114112The number of Unicode code points: scalar values are below it.
SURROGATE_FIRST
const SURROGATE_FIRST = 55296The first surrogate code point, which is not a scalar value.
SURROGATE_LAST
const SURROGATE_LAST = 57343The last surrogate code point.
MAX_VARINT_BYTES
const MAX_VARINT_BYTES = 10The most bytes of one encoded varint.