From 97fd3e249510bccfb246de5667361188ee29cfd7 Mon Sep 17 00:00:00 2001 From: Jonathan Shook Date: Fri, 4 Sep 2026 22:37:28 +0000 Subject: [PATCH] docs(vectordata): describe the records layer in the contract State the record, wire-format, vernacular, and facet-shape parts of the contract, and replace the note that slab records are not decoded with the three deviations that remain: a tree codec in place of serde, document-order JSON parsing, and binding forms out of scope. --- nb-vectordata/README.md | 40 +++++++++++++-- nb-vectordata/RUST_COMPATIBILITY.md | 80 +++++++++++++++++++++++++---- 2 files changed, 107 insertions(+), 13 deletions(-) diff --git a/nb-vectordata/README.md b/nb-vectordata/README.md index 2b6aa050e..2b56830b3 100644 --- a/nb-vectordata/README.md +++ b/nb-vectordata/README.md @@ -2,9 +2,9 @@ `nb-vectordata` is the standalone Java programmatic-access implementation of the vectordata dataset format. It is intentionally limited to catalog, -manifest, vector-reader, transport, cache, and integrity-verification APIs; -workload generators and legacy command-line tooling are not part of this -module. +manifest, vector-reader, record-codec, transport, cache, and +integrity-verification APIs; workload generators and legacy command-line +tooling are not part of this module. The compatibility contract is [RUST_COMPATIBILITY.md](RUST_COMPATIBILITY.md). @@ -56,7 +56,39 @@ cannot map is refused under `REFUSE`, exactly as a requested window would be; `ALLOW` accepts the whole facet instead. Slab facets — paged metadata, addressable as `m.slab` or by namespace as `m.slab:content` — are planned and fetched by the pages a window spans, located through the index in the -slab's tail; this module does not decode their records. +slab's tail, and read by ordinal as records (below). + +## Reading a slab facet's records + +A facet holds either runs of a fixed-width element or opaque records of +their own length. Ask which before choosing a reader; each refuses the +other shape by naming the reader that opens it: + +```java +if (view.facetShape("metadata_content") == FacetShape.RECORDS) { + RecordFacet facet = view.openFacetRecords("metadata_content"); + Records nodes = facet.decode(Codecs.ANODE); // stage 1: the record as it is + Records json = facet.decode(Codecs.text(Vernacular.JSON)); // stage 2: rendered + Records trees = facet.decode(Codecs.TREE); // plain maps and lists + Records byName = facet.decode(Codecs.byName("readout")); + + ANode first = nodes.get(0); + if (first instanceof ANode.M m && view.facetShape("metadata_predicates") == FacetShape.RECORDS) { + ANode p = view.openFacetRecords("metadata_predicates").decode(Codecs.ANODE).get(0); + boolean hit = PredicateEvaluator.evaluate(((ANode.P) p).node(), m.node()); + } + RecordFacet schema = facet.namespace("schema"); // a sibling document, same files +} +``` + +A record decodes by its own leader byte — `0x01` an `MNode`, `0x02` a +`PNode` — so a facet holding both reads without the caller saying which. +Rendering and parsing are available directly through `Vernaculars`, +in every vernacular the reference names (`json`, `jsonl`, `yaml`, `sql`, +`sqlite`, `cql`, `cddl`, `readout`, their `-schema` and `-value` forms, +and `display`); a `PNode`'s display form reads back with `PNodeDisplay`. +Reading one record from a remote facet fetches the page that holds it, +not the file. A series of slabs is one facet of records in shard order. ## Facets spread across several files diff --git a/nb-vectordata/RUST_COMPATIBILITY.md b/nb-vectordata/RUST_COMPATIBILITY.md index b48657210..756ee486b 100644 --- a/nb-vectordata/RUST_COMPATIBILITY.md +++ b/nb-vectordata/RUST_COMPATIBILITY.md @@ -29,8 +29,8 @@ implementation's current behavior and its format tests are normative. and the Rust walk-built form of `N` record starts with no sentinel. Slab containers (`.slab`) — paged records indexed by ordinal through a pages page in the file's tail, with optional named namespaces — are - planned, prefetched, and prebuffered by the pages a window spans; their - records are not decoded here (see below). + planned, prefetched, and prebuffered by the pages a window spans, and + read by ordinal as records (see *Records* below). - Multi-file facets: a facet may be a *series* of files forming one dense, gapless ordinal space, declared in either of the reference forms — uniform (`source: base__NNNN.fvec` with `shard_stride`, `shard_count`, @@ -104,6 +104,57 @@ implementation's current behavior and its format tests are normative. window the format cannot map under `REFUSE` exactly as a requested one — so a sized profile over a large base fetches what it can address and nothing more. `prebuffer(PrebufferProgress)` is the `REFUSE` form. +- Records: a slab facet opens through `openFacetRecords` as a + `RecordFacet` — ordinals and bytes, before a codec is chosen — and a + codec applied with `decode` yields a typed `Records` reader. The facet + holds one container per file, each shard of a series an ordinary slab + based at zero, so a facet ordinal is answered by the container that owns + it at its local ordinal; counts come from the containers, never from + the shard declaration. A namespace written into the declaration + (`m.slab:content`, `namespace:`) selects that document, one absent from + the container holds no records, and `namespace(name)` opens a sibling + document of the same containers. Reading one record from a remote + facet costs one page, not the file. The codecs compose the reference's + two stages: `Codecs.ANODE` stops after stage 1 — the record as the + `ANode` its leader byte says it is, `0x01` an `MNode` and `0x02` a + `PNode`, never inferred from the facet; `Codecs.text(vernacular)` adds + stage 2 and renders it; `Codecs.TREE` hands back plain Java values, the + role a serde target plays in the reference. `Codecs.byName` resolves a + vernacular name through the same table every other by-name surface + uses, so a codec chosen at runtime and one written in code decode + identically; `anode` names none, since it produces no text. +- ANode wire formats: an `MNode` is `[0x01][u16 count]` of + `[u16 name][name][tag][value]` fields over the 29 `TypeTag`s (`text` + through `typed_map`, little-endian, `text_validated` read as text and + `decimal`/`varint` as bytes). A `PNode` is a pre-order tree of + `[conjugate type][…]` nodes in three sub-formats: indexed (`[0][field + index][op][i16 count][i64…]`), named legacy (a conjugate byte after + the leader, `i64` comparands), and named typed (`0xFF` after the + leader, comparands tagged `0` int, `1` float, `2` text, `3` bool, `4` + bytes, `5` null). Fingerprints replace every value with its type + default; congruence compares fingerprints. The `Display` grammar + (`age > 18`, `status IN (1, 2)`, `(a = 1 AND b < 2)`) round-trips + through `PNodeDisplay.parse`. `PredicateEvaluator` applies a tree to a + record with the reference's coercions — integer families against `Int`, + float families against `Float`, the two cross-compared numerically, + text families against `Text`, `MATCHES` as substring containment over + text or bytes, a missing field equal to `NULL` and nothing else. +- Vernaculars: the reference's thirteen — `cddl`, `cddl-value`, `sql`, + `sql-schema`, `sqlite`, `sqlite-schema`, `cql`, `cql-schema`, `json`, + `jsonl`, `yaml`, `readout`, `display` — each rendering both node kinds + in its own syntax, with the sqlite pair spelled as the sql pair. JSON, + YAML, SQL, CQL, CDDL, and readout parse back with the reference's type + inference (quoted strings, bare integers, decimal-point floats, + `true`/`false`, each language's null); the schema, value, and display + renderings refuse to, naming themselves. +- Facet shape: every extension the spec names belongs to a `FacetFormat` + with a `FacetShape` — element runs, or opaque records — and `facetShape` + answers it so a caller handling both branches on the fact rather than + on a failure. Each reader refuses the other shape at the door, naming + the reader that opens it (`WrongFacetShapeException` from the element + readers, `RecordException` of kind `WRONG_SHAPE` from the record path) + instead of failing on the symptom — a slab parsed as vectors, or an + xvec parsed for a footer. - Types: signed and unsigned 8/16/32/64-bit integers plus f16, f32, and f64. - Sources: local files, `file:` URIs, and HTTP(S) URLs. - Remote caching: sparse range caching, `.mref` SHA-256 Merkle verification, @@ -173,13 +224,24 @@ Remaining representation differences: `*vvec` extensions carry variable-length records. (The Rust source's `is_vvec_ext` still classifies `ivec` as variable; to be reconciled upstream.) -- **Slab records are not decoded.** The reference reads slab records - through its ANode codecs and binds them to operation parameters; this - module carries none of that, so `openFacet` on a slab facet fails with - an unsupported extension. Slab facets take part in everything that - does not depend on what a record *is* — windows, shards, residency, - planning, prefetch, and the whole-profile prebuffer — which is what a - sized profile over a slab-backed dataset needs. +- **The serde codec is a tree codec.** The reference's third codec + deserializes into any serde target; Java has no serde, so `Codecs.TREE` + produces the one target that stands for all of them — an + insertion-ordered map of `String`, `Long`, `Double`, `Boolean`, `null`, + lists, and nested maps, typed exactly as the JSON vernacular types the + same record. A caller wanting a class binds it from that map. +- **JSON parses in document order.** The JSON vernacular reads back with + a minimal strict parser that keeps object keys in the order written, so + a record round-tripped through JSON keeps its field order; the reference + relies on its JSON library's map, whose order is that library's choice. + Numbers follow the reference's inference: no point or exponent and fits + a long is an integer, anything else numeric is a float. +- **Record binding forms are not ported.** The reference binds decoded + records to operation parameters and walks MNode fields without + allocating; both belong to workload generation, not to the access API, + and are out of this module's scope. Slab facets take part in everything + else — windows, shards, residency, planning, prefetch, the + whole-profile prebuffer, and now records by ordinal. - **A windowed reader's `prebuffer` fetches its window.** The reference's windowed reader inherits a no-op `precache`; here the reader a view hands back for a windowed facet warms the same bytes the whole-profile prebuffer