ocaml-catalog/test/ at main · gazagnaire.org/ocaml-catalog · Tangled

gazagnaire.org / ocaml-catalog

0

Storage-agnostic SQL query engine over a pluggable catalog SPI

0

ocaml-catalog / test

13 files

at main

Thomas Gazagnaire catalog: compile a parameter-independent VDBE program 4w ago

catalog: compile a parameter-independent VDBE program A bound [?] compiled to the literal value of the parameter (operand baked params.(i) in via emit_value), so the VM program was specific to one binding and could not be reused across executions with different parameters -- every run recompiled. Compile [?] to a new Load_param opcode instead: Vdbe.run takes the bound values as registers and Load_param copies param i into its destination at run time (an out-of-range index reads NULL, as an unbound ? does in sqlite3). Ir.compile no longer takes the parameters and the whole expr-emit chain drops them; bindings are supplied to Vdbe.run via the new Ir.mems_of_params. The program is now a pure function of (SQL, schema) -- the basis for a plan cache. Behaviour is unchanged: the 60 catalog and 28 sqlite tests stay green.

4 weeks ago

catalog: typed query and write layer as the default API Make the typed surface the front door of the catalog package and move the stringly backend contract under Catalog.Spi (the service provider interface a store implements), now writable: insert/update/delete in the shape of SQLite's virtual-table xUpdate and PostgreSQL's writable foreign-data wrappers. A row to mutate is addressed by what rows_rowid reported -- rowid plus old values -- so a numbered store keys on the rowid and a keyed store (an Irmin tree) on the key column. Mem grows stable rowids and the full write surface. A table is declared once as a row codec -- typed columns assembled into a record constructor and sealed, like a nox-json object map -- and the same codec drives queries and writes. Queries are comprehensions in the Links/T-LINQ spirit: from/join/left_join bind one variable per table (the binder is the alias, so self-joins are free and out-of-scope columns are unrepresentable), projections and grouped projections are assembled and sealed like object maps, and the grouped constructor only ever receives keys and aggregates, so projecting an ungrouped column is a type error. Left joins are row-level options decoded through a match witness, so a no-match is None even when a matched row is NULL in every selected column. Execution lowers to the Ir engine; equality predicates are offered to the backend's index_scan (explain prints SCAN/SEARCH ... USING INDEX). Two engine edges close: Vdbe gains a blob-literal load and Ir a Matched expr reading the left-join match flag. Tests: a functional suite over Mem plus an indexed fixture backend, the runtime failures the types cannot rule out, and an MDX typing.md that pins the exact compiler errors for ill-typed queries so a refactor that weakens a static guarantee fails dune test.

1 month ago

test_catalog.ml

tree: reformat for the upgraded ocamlformat Comment reflow and header reindent produced by the new ocamlformat; no code changes.

1 month ago

test_catalog.mli

catalog: typed query and write layer as the default API Make the typed surface the front door of the catalog package and move the stringly backend contract under Catalog.Spi (the service provider interface a store implements), now writable: insert/update/delete in the shape of SQLite's virtual-table xUpdate and PostgreSQL's writable foreign-data wrappers. A row to mutate is addressed by what rows_rowid reported -- rowid plus old values -- so a numbered store keys on the rowid and a keyed store (an Irmin tree) on the key column. Mem grows stable rowids and the full write surface. A table is declared once as a row codec -- typed columns assembled into a record constructor and sealed, like a nox-json object map -- and the same codec drives queries and writes. Queries are comprehensions in the Links/T-LINQ spirit: from/join/left_join bind one variable per table (the binder is the alias, so self-joins are free and out-of-scope columns are unrepresentable), projections and grouped projections are assembled and sealed like object maps, and the grouped constructor only ever receives keys and aggregates, so projecting an ungrouped column is a type error. Left joins are row-level options decoded through a match witness, so a no-match is None even when a matched row is NULL in every selected column. Execution lowers to the Ir engine; equality predicates are offered to the backend's index_scan (explain prints SCAN/SEARCH ... USING INDEX). Two engine edges close: Vdbe gains a blob-literal load and Ir a Matched expr reading the left-join match flag. Tests: a functional suite over Mem plus an indexed fixture backend, the runtime failures the types cannot rule out, and an MDX typing.md that pins the exact compiler errors for ill-typed queries so a refactor that weakens a static guarantee fails dune test.

1 month ago

catalog: compile a parameter-independent VDBE program A bound [?] compiled to the literal value of the parameter (operand baked params.(i) in via emit_value), so the VM program was specific to one binding and could not be reused across executions with different parameters -- every run recompiled. Compile [?] to a new Load_param opcode instead: Vdbe.run takes the bound values as registers and Load_param copies param i into its destination at run time (an out-of-range index reads NULL, as an unbound ? does in sqlite3). Ir.compile no longer takes the parameters and the whole expr-emit chain drops them; bindings are supplied to Vdbe.run via the new Ir.mems_of_params. The program is now a pure function of (SQL, schema) -- the basis for a plan cache. Behaviour is unchanged: the 60 catalog and 28 sqlite tests stay green.

4 weeks ago

catalog: a positional query IR and its compiler to the VM (engine transition step 1) This begins moving the SQL engine into ocaml-catalog so any backend runs it and the tree-walker can eventually be deleted there. The blocker has been that the SQL AST aliases the B-tree value type, so it cannot move into catalog; the seam is a positional, alias-free IR over Catalog.Value that the compiler turns into a Vdbe.program, with the frontend lowering its AST to the IR (resolving names to (cursor,column) positions, literals to Value.t) at the boundary. Catalog.Ir models a single-cursor scan / filter / project -- expressions (column/literal/param/arithmetic), an AND of comparisons and IS [NOT] NULL, and a projection -- and compiles them to the register VM (label resolution included), storage- and dialect-agnostic. Tested end-to-end over a list_cursor: scan/filter/project, AND + IS NOT NULL, and a bound parameter. Next on this seam (parallelisable): joins (nested loops), the injected scalar function/value layer, aggregation/window post-passes, and the Ast->Ir lowering in ocaml-sqlite that lets it call this compiler instead of vdbe_run.

1 month ago

docs: reflow comments across the tree

1 month ago

catalog: typed query and write layer as the default API Make the typed surface the front door of the catalog package and move the stringly backend contract under Catalog.Spi (the service provider interface a store implements), now writable: insert/update/delete in the shape of SQLite's virtual-table xUpdate and PostgreSQL's writable foreign-data wrappers. A row to mutate is addressed by what rows_rowid reported -- rowid plus old values -- so a numbered store keys on the rowid and a keyed store (an Irmin tree) on the key column. Mem grows stable rowids and the full write surface. A table is declared once as a row codec -- typed columns assembled into a record constructor and sealed, like a nox-json object map -- and the same codec drives queries and writes. Queries are comprehensions in the Links/T-LINQ spirit: from/join/left_join bind one variable per table (the binder is the alias, so self-joins are free and out-of-scope columns are unrepresentable), projections and grouped projections are assembled and sealed like object maps, and the grouped constructor only ever receives keys and aggregates, so projecting an ungrouped column is a type error. Left joins are row-level options decoded through a match witness, so a no-match is None even when a matched row is NULL in every selected column. Execution lowers to the Ir engine; equality predicates are offered to the backend's index_scan (explain prints SCAN/SEARCH ... USING INDEX). Two engine edges close: Vdbe gains a blob-literal load and Ir a Matched expr reading the left-join match flag. Tests: a functional suite over Mem plus an indexed fixture backend, the runtime failures the types cannot rule out, and an MDX typing.md that pins the exact compiler errors for ill-typed queries so a refactor that weakens a static guarantee fails dune test.

1 month ago

tree: reformat for the upgraded ocamlformat Comment reflow and header reindent produced by the new ocamlformat; no code changes.

1 month ago

catalog: storage-agnostic SQL backend SPI (keystone) New ocaml-catalog package -- the seam a generic SQL engine runs over, so a B-tree, a Parquet file, or an Irmin tree becomes SQL-queryable by presenting a catalog, without duplicating the engine (cf. SQLite vtab / SQL-MED FDW / DataFusion TableProvider). - Catalog.Value: the five SQLite storage classes (Null/Int/Float/Blob/ Text) + compare (storage-class order), equal, to_text. - Catalog.t: abstract, flat accessors (columns/rows/rows_rowid/index_scan/ index_for/collation); with_table derives a catalog for CTEs/subqueries. - module type S (functor-friendly backend SPI over a handle) + Make (B : S) : sig val v : B.handle -> t end. - Catalog.Mem reference in-memory backend + of_mem. Phase 1 of factoring the engine out of ocaml-sqlite. 6 tests, merlint clean.

1 month ago

tree: reformat for the upgraded ocamlformat Comment reflow and header reindent produced by the new ocamlformat; no code changes.

1 month ago

catalog: lift the register VM (Vdbe) into the engine core The SQLite-shaped register VM -- mutable native-int mem cells reused per row, opcode array, cursor abstraction, O(registers)-per-scan allocation -- is the engine's execution substrate, not something specific to the SQL frontend or the b-tree store. It uses only Stdlib (no AST, no storage format), so it lifts into catalog unchanged and becomes the shared core every backend's cursor feeds. ocaml-sqlite will re-export it next so there is one implementation. The VM's SQL-free contract -- a hand-built program over a hand-written cursor, plus the O(1)-allocation-per-row invariant -- comes along as catalog's own test.

1 month ago

catalog: typed query and write layer as the default API Make the typed surface the front door of the catalog package and move the stringly backend contract under Catalog.Spi (the service provider interface a store implements), now writable: insert/update/delete in the shape of SQLite's virtual-table xUpdate and PostgreSQL's writable foreign-data wrappers. A row to mutate is addressed by what rows_rowid reported -- rowid plus old values -- so a numbered store keys on the rowid and a keyed store (an Irmin tree) on the key column. Mem grows stable rowids and the full write surface. A table is declared once as a row codec -- typed columns assembled into a record constructor and sealed, like a nox-json object map -- and the same codec drives queries and writes. Queries are comprehensions in the Links/T-LINQ spirit: from/join/left_join bind one variable per table (the binder is the alias, so self-joins are free and out-of-scope columns are unrepresentable), projections and grouped projections are assembled and sealed like object maps, and the grouped constructor only ever receives keys and aggregates, so projecting an ungrouped column is a type error. Left joins are row-level options decoded through a match witness, so a no-match is None even when a matched row is NULL in every selected column. Execution lowers to the Ir engine; equality predicates are offered to the backend's index_scan (explain prints SCAN/SEARCH ... USING INDEX). Two engine edges close: Vdbe gains a blob-literal load and Ir a Matched expr reading the left-join match flag. Tests: a functional suite over Mem plus an indexed fixture backend, the runtime failures the types cannot rule out, and an MDX typing.md that pins the exact compiler errors for ill-typed queries so a refactor that weakens a static guarantee fails dune test.

1 month ago