Skip to main content

IR specification

The canonical IR (pkg/ir) is the contract between query-language frontends and the LynxDB engine. A plan is a tree of logical operators (ir.Node); scalar computations are typed expressions (ir.Expr). The IR names semantic database operations, not the syntax of any language, and it does not depend on storage formats.

Types​

IR typeValues
anyA value whose type is decided per row (schema-on-read columns).
nullThe type of the NULL literal; unifies with every type.
boolTRUE, FALSE.
int64-bit signed integer.
float64-bit IEEE 754 float.
stringUTF-8 text.
timestampAn instant with nanosecond precision, UTC.
durationA signed span of time; also used for times of day.
array<T>Ordered list.
objectString-keyed map.

Unify gives the common type of CASE branches, COALESCE arguments and UNION columns: equal types stay, NULL yields the other type, int and float widen to float, everything else becomes any. A float parameter accepts an int argument; any and null arguments are accepted everywhere and checked at run time.

Schemas​

Every operator exposes its output schema: columns with name, type, nullability and optional metadata. A closed schema lists every column; an open schema lists the columns known at planning time and allows further columns of type any.

NULL, missing and three-valued logic​

  • A reference to a column absent from a row evaluates to NULL. IsMissing(col) is TRUE only for an absent column; is_null(x) is TRUE for NULL or absent.
  • Comparisons and arithmetic with a NULL operand yield NULL.
  • AND, OR, XOR and NOT use Kleene logic.
  • Filter keeps a row only when its condition is TRUE.
  • InList is TRUE on a match, NULL when there is no match and the tested value or an element is NULL, FALSE otherwise.
  • Integer division and modulo by zero yield NULL.

Run-time conversions​

Values of type any meet each other at run time. The binary operators apply these rules:

  • int with int is int; int with float is float.
  • Adding two strings concatenates them.
  • A string meeting a number in arithmetic or a comparison is converted to a number; it yields NULL when it does not parse.
  • A string compared with a timestamp is converted to a timestamp; it yields NULL when it does not parse.
  • timestamp - timestamp is a duration; timestamp ± duration is a timestamp.
  • Other incompatible combinations yield NULL.

Cast converts explicitly; a value that cannot be converted yields NULL. Casting a float to int truncates toward zero; casting a string to bool accepts true and false in any case; casting a number to a timestamp reads epoch seconds.

Expressions​

ExpressionMeaning
LiteralA constant.
FieldRefA column; a dotted name reads the column with exactly that name, else walks object members.
Unary, BinaryNegation, NOT; arithmetic, comparisons, AND, OR, XOR.
CallA registered scalar function.
CaseFirst branch whose condition is TRUE, else the ELSE value or NULL.
CastExplicit conversion.
ArrayExpr, ObjectExpr, IndexConstruction and element access.
Lambda, LambdaParamAnonymous functions, only as arguments of higher-order functions.
InListMembership test.
IsMissingColumn absence.
RelativeTimeAn instant relative to the query's reference time (now-1d@d).
SearchText predicate; see below.
SubqueryUncorrelated EXISTS, IN or scalar subquery.
AggCallAggregate or window function application, only in Aggregate and Window.

Search predicates​

Search{Mode, Field, Pattern, CaseSensitive} keeps the distinctions that let a scan leave out data. Field empty means a free-text search of the whole event.

ModeMatchesColumnar executor
wordevery token of the pattern as a whole tokentoken filters, then per row
prefixa token starting with the patternper row
globa token matching */? wildcardsa single trailing * as a prefix; other patterns run on the row operators
phrasethe pattern at token boundariestoken filters on the phrase's tokens, then per row
substringthe pattern anywhereper row
regexan RE2 expressionper row

Operators​

OperatorSemantics
ScanReads events of sources (names or * patterns) within an exact time range. Carries annotations: pruning predicates, required columns, a row limit, descending order. Annotations never change results.
ValuesLiteral rows.
EmptyNo rows.
SourceSchemaOne row per column of a source.
FilterKeeps rows whose condition is TRUE.
ProjectKeeps (in listed order) or drops columns; names may contain *.
RenameColumnsRenames columns, with * captures.
ComputeAdds or replaces columns; assignments see earlier ones.
AggregateGroups by expressions and computes aggregates; output is groups then aggregates; one row without groups. Rows without a group value form a group whose key is NULL; DropNullGroups drops them.
WindowAggregates per row over the whole partition or a running frame (row bound, current-row exclusion).
Sort, TopN, Limit, ReverseOrdering with explicit NULL placement; bounded heap top-N; offset and count; reversal.
DedupFirst N rows per key, optional NULL keys, optional consecutive mode.
Joininner, left, right, full, semi, anti, cross; equality keys plus residual condition; per-row match cap; collision rule; optional qualified copies of each side's columns.
UnionConcatenation by column name, or newest-first merge.
ZipColumnsPositional pairing of two inputs.
ExtractRegex captures or JSON members from a string expression.
PatternsLog pattern labeling or aggregation.
UnnestOne row per array element.
FlattenObjectObject members as columns.
FillNullNULL replacement per column or for all columns.
Transpose, Totals, CombineValuesReshaping operators.

Function registry​

ir.Funcs() lists every function with its kind (scalar, aggregate, window), category, parameters (type, constant-only, lambda arity), result rule, NULL behaviour (strict, aware, skipped), volatility (volatile, query constant), merge kind for partial aggregation (exact, approximate, none), order sensitivity and documentation. Foldable functions with constant arguments are folded at planning time. Every scalar function has a VM implementation; pkg/colstore/exec/irfunc implements them for the columnar executor with the same semantics.

Validation​

ir.Validate checks operator shapes, column resolution against closed input schemas, function kinds, arity and static argument types, constant arguments, aggregate placement, lambda scoping and subquery shapes.

Optimizer​

pkg/ir/opt applies, to a fixed point:

  • constant folding of deterministic calls, through an evaluator the caller supplies;
  • merging adjacent filters and removing TRUE filters;
  • copying indexable conjuncts (free-text searches, column equals string, disjunctions of words) into Scan.Prune, keeping the filter;
  • narrowing Scan.Time with constant comparisons of the time column;
  • Sort + Limit into TopN;
  • pushing a row bound into Scan.Limit through Compute, Project and RenameColumns;
  • recording the required columns on each scan.

See IR optimizer for each rule.

Execution​

The IR plan is what both executors run. The columnar executor (pkg/colstore/exec/irplan translates the plan) takes the chain of operators from a scan upwards that it can run; the row operators (pkg/engine/physical) run the rest. See Pipeline execution.

physical.PlanHints (pkg/engine/physical/hints.go) summarizes the scans of an optimized plan: sources, the resolved time range, search terms and token globs from free-text Search pruning predicates, field predicates for fields equal to strings, required columns, the row limit and the scan order. A plan with one scan gets all of them; a plan with several scans gets only the union of their sources and time ranges. GET /api/v1/query/explain reports this summary.