docs(vectordata): describe the records layer in the contract

State the record, wire-format, vernacular, and facet-shape parts of the
contract, and replace the note that slab records are not decoded with
the three deviations that remain: a tree codec in place of serde,
document-order JSON parsing, and binding forms out of scope.
This commit is contained in:
Jonathan Shook committed 2026-09-04 22:37:28 +00:00
1 parent e315c4bca5
commit 97fd3e2495
2 files changed
+107 -13

No files matched your search

+36 -4
View File
@@ -2,9 +2,9 @@
`nb-vectordata` is the standalone Java programmatic-access implementation of
the vectordata dataset format. It is intentionally limited to catalog,
manifest, vector-reader, transport, cache, and integrity-verification APIs;
workload generators and legacy command-line tooling are not part of this
module.
manifest, vector-reader, record-codec, transport, cache, and
integrity-verification APIs; workload generators and legacy command-line
tooling are not part of this module.
The compatibility contract is [RUST_COMPATIBILITY.md](RUST_COMPATIBILITY.md).
@@ -56,7 +56,39 @@ cannot map is refused under `REFUSE`, exactly as a requested window would be;
`ALLOW` accepts the whole facet instead. Slab facets — paged metadata,
addressable as `m.slab` or by namespace as `m.slab:content` — are planned
and fetched by the pages a window spans, located through the index in the
slab's tail; this module does not decode their records.
slab's tail, and read by ordinal as records (below).
## Reading a slab facet's records
A facet holds either runs of a fixed-width element or opaque records of
their own length. Ask which before choosing a reader; each refuses the
other shape by naming the reader that opens it:
```java
if (view.facetShape("metadata_content") == FacetShape.RECORDS) {
RecordFacet facet = view.openFacetRecords("metadata_content");
Records<ANode> nodes = facet.decode(Codecs.ANODE); // stage 1: the record as it is
Records<String> json = facet.decode(Codecs.text(Vernacular.JSON)); // stage 2: rendered
Records<Object> trees = facet.decode(Codecs.TREE); // plain maps and lists
Records<String> byName = facet.decode(Codecs.byName("readout"));
ANode first = nodes.get(0);
if (first instanceof ANode.M m && view.facetShape("metadata_predicates") == FacetShape.RECORDS) {
ANode p = view.openFacetRecords("metadata_predicates").decode(Codecs.ANODE).get(0);
boolean hit = PredicateEvaluator.evaluate(((ANode.P) p).node(), m.node());
}
RecordFacet schema = facet.namespace("schema"); // a sibling document, same files
}
```
A record decodes by its own leader byte — `0x01` an `MNode`, `0x02` a
`PNode` — so a facet holding both reads without the caller saying which.
Rendering and parsing are available directly through `Vernaculars`,
in every vernacular the reference names (`json`, `jsonl`, `yaml`, `sql`,
`sqlite`, `cql`, `cddl`, `readout`, their `-schema` and `-value` forms,
and `display`); a `PNode`'s display form reads back with `PNodeDisplay`.
Reading one record from a remote facet fetches the page that holds it,
not the file. A series of slabs is one facet of records in shard order.
## Facets spread across several files
+71 -9
View File
@@ -29,8 +29,8 @@ implementation's current behavior and its format tests are normative.
and the Rust walk-built form of `N` record starts with no sentinel.
Slab containers (`.slab`) — paged records indexed by ordinal through a
pages page in the file's tail, with optional named namespaces — are
planned, prefetched, and prebuffered by the pages a window spans; their
records are not decoded here (see below).
planned, prefetched, and prebuffered by the pages a window spans, and
read by ordinal as records (see *Records* below).
- Multi-file facets: a facet may be a *series* of files forming one dense,
gapless ordinal space, declared in either of the reference forms —
uniform (`source: base__NNNN.fvec` with `shard_stride`, `shard_count`,
@@ -104,6 +104,57 @@ implementation's current behavior and its format tests are normative.
window the format cannot map under `REFUSE` exactly as a requested one —
so a sized profile over a large base fetches what it can address and
nothing more. `prebuffer(PrebufferProgress)` is the `REFUSE` form.
- Records: a slab facet opens through `openFacetRecords` as a
`RecordFacet` — ordinals and bytes, before a codec is chosen — and a
codec applied with `decode` yields a typed `Records` reader. The facet
holds one container per file, each shard of a series an ordinary slab
based at zero, so a facet ordinal is answered by the container that owns
it at its local ordinal; counts come from the containers, never from
the shard declaration. A namespace written into the declaration
(`m.slab:content`, `namespace:`) selects that document, one absent from
the container holds no records, and `namespace(name)` opens a sibling
document of the same containers. Reading one record from a remote
facet costs one page, not the file. The codecs compose the reference's
two stages: `Codecs.ANODE` stops after stage 1 — the record as the
`ANode` its leader byte says it is, `0x01` an `MNode` and `0x02` a
`PNode`, never inferred from the facet; `Codecs.text(vernacular)` adds
stage 2 and renders it; `Codecs.TREE` hands back plain Java values, the
role a serde target plays in the reference. `Codecs.byName` resolves a
vernacular name through the same table every other by-name surface
uses, so a codec chosen at runtime and one written in code decode
identically; `anode` names none, since it produces no text.
- ANode wire formats: an `MNode` is `[0x01][u16 count]` of
`[u16 name][name][tag][value]` fields over the 29 `TypeTag`s (`text`
through `typed_map`, little-endian, `text_validated` read as text and
`decimal`/`varint` as bytes). A `PNode` is a pre-order tree of
`[conjugate type][…]` nodes in three sub-formats: indexed (`[0][field
index][op][i16 count][i64…]`), named legacy (a conjugate byte after
the leader, `i64` comparands), and named typed (`0xFF` after the
leader, comparands tagged `0` int, `1` float, `2` text, `3` bool, `4`
bytes, `5` null). Fingerprints replace every value with its type
default; congruence compares fingerprints. The `Display` grammar
(`age > 18`, `status IN (1, 2)`, `(a = 1 AND b < 2)`) round-trips
through `PNodeDisplay.parse`. `PredicateEvaluator` applies a tree to a
record with the reference's coercions — integer families against `Int`,
float families against `Float`, the two cross-compared numerically,
text families against `Text`, `MATCHES` as substring containment over
text or bytes, a missing field equal to `NULL` and nothing else.
- Vernaculars: the reference's thirteen — `cddl`, `cddl-value`, `sql`,
`sql-schema`, `sqlite`, `sqlite-schema`, `cql`, `cql-schema`, `json`,
`jsonl`, `yaml`, `readout`, `display` — each rendering both node kinds
in its own syntax, with the sqlite pair spelled as the sql pair. JSON,
YAML, SQL, CQL, CDDL, and readout parse back with the reference's type
inference (quoted strings, bare integers, decimal-point floats,
`true`/`false`, each language's null); the schema, value, and display
renderings refuse to, naming themselves.
- Facet shape: every extension the spec names belongs to a `FacetFormat`
with a `FacetShape` — element runs, or opaque records — and `facetShape`
answers it so a caller handling both branches on the fact rather than
on a failure. Each reader refuses the other shape at the door, naming
the reader that opens it (`WrongFacetShapeException` from the element
readers, `RecordException` of kind `WRONG_SHAPE` from the record path)
instead of failing on the symptom — a slab parsed as vectors, or an
xvec parsed for a footer.
- Types: signed and unsigned 8/16/32/64-bit integers plus f16, f32, and f64.
- Sources: local files, `file:` URIs, and HTTP(S) URLs.
- Remote caching: sparse range caching, `.mref` SHA-256 Merkle verification,
@@ -173,13 +224,24 @@ Remaining representation differences:
`*vvec` extensions carry variable-length records. (The Rust source's
`is_vvec_ext` still classifies `ivec` as variable; to be reconciled
upstream.)
- **Slab records are not decoded.** The reference reads slab records
through its ANode codecs and binds them to operation parameters; this
module carries none of that, so `openFacet` on a slab facet fails with
an unsupported extension. Slab facets take part in everything that
does not depend on what a record *is* — windows, shards, residency,
planning, prefetch, and the whole-profile prebuffer — which is what a
sized profile over a slab-backed dataset needs.
- **The serde codec is a tree codec.** The reference's third codec
deserializes into any serde target; Java has no serde, so `Codecs.TREE`
produces the one target that stands for all of them — an
insertion-ordered map of `String`, `Long`, `Double`, `Boolean`, `null`,
lists, and nested maps, typed exactly as the JSON vernacular types the
same record. A caller wanting a class binds it from that map.
- **JSON parses in document order.** The JSON vernacular reads back with
a minimal strict parser that keeps object keys in the order written, so
a record round-tripped through JSON keeps its field order; the reference
relies on its JSON library's map, whose order is that library's choice.
Numbers follow the reference's inference: no point or exponent and fits
a long is an integer, anything else numeric is a float.
- **Record binding forms are not ported.** The reference binds decoded
records to operation parameters and walks MNode fields without
allocating; both belong to workload generation, not to the access API,
and are out of this module's scope. Slab facets take part in everything
else — windows, shards, residency, planning, prefetch, the
whole-profile prebuffer, and now records by ordinal.
- **A windowed reader's `prebuffer` fetches its window.** The reference's
windowed reader inherits a no-op `precache`; here the reader a view hands
back for a windowed facet warms the same bytes the whole-profile prebuffer