GizStack
View .mdOpen in ChatGPTOpen in Claude

gizmodel/sql/lex

SQL tokens. One lexer for every dialect: identifiers bare, "quoted", [bracketed] (SQL Server) or backquoted (MySQL); strings '...' with '' inside, E'...' escapes and $tag$...$tag$ (PostgreSQL) and N'...' (SQL Server, Oracle); numbers; parameters $1, ?, ?1, :name and @name; operators; and comments -- and /* */.

gizlang
import * as Lex from "gizmodel/sql/lex";

Types

Token

gizlang
type Token = { kind: i64; text: string; at: i64; }

A token: its kind, its text, and its byte offset in the source.

  • kind: i64
  • text: string
  • at: i64

Literals

gizlang
type Literals = { text: string; values: Vec<string>; types: Vec<i64>; }

A text with its compared literals as parameters (comparedLiterals): text has $1, $2, ... where they were, values are their texts and types their PostgreSQL types: 0 (unknown) for a string, as a quoted literal is, int4 or int8 for an integer.

  • text: string
  • values: Vec<string>
  • types: Vec<i64>

Functions

keyword

gizlang
function keyword(token: Token, word: string): bool

An unquoted ASCII keyword, compared without allocating a lowercase token.

Effects: fail.

skipParens

gizlang
function skipParens(toks: Vec<Token>, at: i64): i64

The index after the parenthesized group that starts at token at, or the END token if it is not closed.

Effects: fail.

userOperator

gizlang
function userOperator(t: string): bool

Whether an operator token is none of the built-in ones the lexer knows: a user-defined operator (parse.giz).

Effects: fail, alloc.

lex

gizlang
function lex(sql: string): Vec<Token> throws string

The tokens of sql, ending with one END token, or the position and reason of the first malformed one.

Effects: fail, alloc.

valueAs

gizlang
function valueAs(check: string, replacement: string): string

A domain check on VALUE with each VALUE as replacement (a quoted column, or a parenthesized cast).

Effects: fail, alloc.

domainChecks

gizlang
function domainChecks(domain: string, text: string): Vec<string>

A domain's CHECK constraints as "name\u0003check" (the domain object's fifth part, "\u0004" between; before, a fourth part held one unnamed check, DOMAIN_check).

Effects: fail, alloc.

comparedLiterals

gizlang
function comparedLiterals(sql: string): Option<Literals>

The literals that a comparison takes in SQL (k = 42, name = 'x') as parameters, so texts that differ only in them share one parsed statement and one shape (psql's and psycopg2's queries carry their values in the text). Only a plain '...' string and an integer, after = < > <= >= <> or !=, and before the end, a word, ")", "," or ";". None when the text has none, or has parameters of its own.

Effects: fail, alloc.

positionalBinds

gizlang
function positionalBinds(sql: string, count: i64): string

SQL with each :name bind written as $N. When COUNT is the number of distinct names, a name repeated keeps one number; else each occurrence takes the next number. Quotes and comments are kept.

Effects: fail, alloc.

engineNameOf

gizlang
function engineNameOf(text: string, dialect: string): string

The engine's name for the quoted identifier text: Oracle keeps a name with a lowercase letter, PostgreSQL one with an uppercase letter, in their encoded form; the rest fold to the unquoted spelling and keep their text; so do the engine's own names.

Effects: fail, alloc.

foldQuoted

gizlang
function foldQuoted(sql: string, dialect: string): string

sql with each double-quoted identifier that dialect folds differently from its text spelled as encodedName. Other dialects, a text without quotes and an unlexable text come back unchanged; an encoded name is left alone.

Effects: fail, alloc.

Constants

IDENT

gizlang
const IDENT: i64 = 1

Token kind: an identifier or a keyword.

QUOTED

gizlang
const QUOTED: i64 = 2

A quoted identifier: never a keyword, case kept.

STRING

gizlang
const STRING: i64 = 3

Token kind: a string literal.

NUMBER

gizlang
const NUMBER: i64 = 4

Token kind: a number.

PARAM

gizlang
const PARAM: i64 = 5

Token kind: a parameter.

OP

gizlang
const OP: i64 = 6

Token kind: an operator or punctuation.

END

gizlang
const END: i64 = 7

Token kind: the end of the text.

BLOB

gizlang
const BLOB: i64 = 8

X'...' blob literal, its text the hex digits.